Jamf Composer has always had two formats to build installers. The standard pkg and the seemlingly standard (but not) dmg. The pkg option will build a standard pkg installer file, which will install with any system that can install pkg files.
The dmg option will build a standard dmg disk image file, with the payload of the installer as contents. On its own, however, this dmg cannot do anything. The Jamf Pro management system how ever will understand what to do and how to install the files from the dmg to a system. There are certain features in Jamf Pro which can install and distribute files to user directories and templates (called ‘Fill User Templates’ FUT and ‘Fill Every User’ FEU) which only work with dmg installers in Jamf Pro.
However, Jamf themselves have been recommending to use the standard pkg format in favor of their proprietary use of dmg. Also the Composer application is 32-bit and its future is uncertain.
Luckily there are plenty of great other third-party tools to build installer packages. I cover many of them in my book: Packaging for Apple Administrators
In general, it is probably preferable to re-visit your imaging process and rebuild any installer you still may have in dmg format from scratch. However, in some cases that might not be possible or necessary.
Since the Composer generated dmgs contain all the files for the payload in the proper folder structure you can just use the entire mounted volume as your payload root for pkgbuild. You can easily convert a Composer generated installer dmg to a standard pkg with these commands:
1) mount the dmg:
$ hdiutil attach /path/to/Sample.dmg
this will output a bunch of info, the very last bit is the mount point of the dmg /Volumes/Sample (the name will depend on the dmg)
2) build a pkg with the contents of the mounted dmg as a payload:
$ pkgbuild --root /Volumes/Sample --version 1.0 --identifier com.example.sample --install-location / Sample-1.0.pkg
This will create Sample-1.0.pkg in your current working directory. (I like to include the version in the pkg file name, but that is entirely optional.)
3) cleanup: unmount the dmg
$ hdiutil detach /Volumes/Sample
Obviously this will not work well with other dmgs, such as Full System dmgs, or dmgs downloaded from the web, which contain an app that should be dragged to /Applications to install (use quickpkg for those dmgs).