My PR needs input from drill developers. Please look for TODO and FIXME in this PR and help me get to where I can initialize this plugin.
In general I copied things from format-xml contrib, but then took ideas from Json. I was unable to figure out how initialization works from the Excel plugin. The metadata bridge is here, and a stub of the data bridge - handles only simple type "INT" right now, and of course doesn't compile yet. https://github.com/apache/drill/pull/2836 ---------- Forwarded message --------- From: Mike Beckerle <notificati...@github.com> Date: Fri, Oct 13, 2023 at 11:11 PM Subject: [apache/drill] WIP: Preliminary Review on adding Daffodil to Drill (PR #2836) To: apache/drill <dr...@noreply.github.com> Cc: Mike Beckerle <mbecke...@apache.org>, Your activity < your_activ...@noreply.github.com> DRILL-2835 <https://issues.apache.org/jira/browse/DRILL-2835>: Preliminary Review on adding Daffodil to Drill Description New format-daffodil module created. But I need assistance with several aspects. Tests of creating Drill schemas from DFDL working. They're simple, but it's showing promise. There are major TODO/FIXME/TBDs in here. Search for FIXME, and "Then a MIRACLE occurs..." This does not compile yet because of the plugin system and how to initialize things. This is the main open problem to get it to compile without error. Needs review by Drill-devs. Documentation TBD: This will require doc eventually Testing Needs more. This is just a preliminary design review Work-in-progress. ------------------------------ You can view, comment on, or merge this pull request online at: https://github.com/apache/drill/pull/2836 Commit Summary - 0633fdb <https://github.com/apache/drill/pull/2836/commits/0633fdbe61bc073cdc1b4fb81551829ff83ecc6c> Checkpoint on adding Daffodil to Drill File Changes (25 files <https://github.com/apache/drill/pull/2836/files>) - *A* contrib/format-daffodil/.gitignore <https://github.com/apache/drill/pull/2836/files#diff-0a627354a4cea5c437f263321497755e8fa3ce7ee4b6e626fee2f3583e1d0416> (2) - *A* contrib/format-daffodil/README.md <https://github.com/apache/drill/pull/2836/files#diff-e96a4fb1ae8531d3b3b3d2aae1611521d2866633e231bf753f958399a6fdd795> (41) - *A* contrib/format-daffodil/pom.xml <https://github.com/apache/drill/pull/2836/files#diff-5c8e06605ea68a8079bd3cb46a8798a530dba2dd21502c546e85dfec94261505> (89) - *A* contrib/format-daffodil/src/main/java/org/apache/drill/exec/store/daffodil/DaffodilBatchReader.java <https://github.com/apache/drill/pull/2836/files#diff-162d1de445d6df5a93c0f3e8aec31db247451e4ff96ec397cfabca3352f1a672> (180) - *A* contrib/format-daffodil/src/main/java/org/apache/drill/exec/store/daffodil/DaffodilDrillInfosetOutputter.java <https://github.com/apache/drill/pull/2836/files#diff-1fe59684b38eb8da600975b5eb3f2d4f4ab4f87bc813ca418ea91f58336be223> (105) - *A* contrib/format-daffodil/src/main/java/org/apache/drill/exec/store/daffodil/DaffodilFormatConfig.java <https://github.com/apache/drill/pull/2836/files#diff-72e26d5dc31d50edbdd44fb7c165e215cfeed2d58e610d3c7f0621f1a9832d77> (97) - *A* contrib/format-daffodil/src/main/java/org/apache/drill/exec/store/daffodil/DaffodilFormatPlugin.java <https://github.com/apache/drill/pull/2836/files#diff-10ba98388e809944fd5f95092d3675c84860d3102c7c9e3237debc4d70c93321> (87) - *A* contrib/format-daffodil/src/main/java/org/apache/drill/exec/store/daffodil/DaffodilMessageParser.java <https://github.com/apache/drill/pull/2836/files#diff-f886e2832e2f734876a672d095fae5004c7c58ee92db5b27623189bba3bc8eaf> (187) - *A* contrib/format-daffodil/src/main/java/org/apache/drill/exec/store/daffodil/schema/DaffodilDataProcessorFactory.java <https://github.com/apache/drill/pull/2836/files#diff-cef9515d31cdfd518179f5ec8d81f31c96d10eb31ef9d138f0d6696a838f01fc> (130) - *A* contrib/format-daffodil/src/main/java/org/apache/drill/exec/store/daffodil/schema/DrillDaffodilSchemaUtils.java <https://github.com/apache/drill/pull/2836/files#diff-c250676beeb4499b8ab42cfa027379a0419290f0d2a2024dc21b213607b28796> (107) - *A* contrib/format-daffodil/src/main/java/org/apache/drill/exec/store/daffodil/schema/DrillDaffodilSchemaVisitor.java <https://github.com/apache/drill/pull/2836/files#diff-31b55c328a69af81ef7e2a4a5794dc902573ed52893d4179bb07020f224ce969> (121) - *A* contrib/format-daffodil/src/main/resources/bootstrap-format-plugins.json <https://github.com/apache/drill/pull/2836/files#diff-bd164b495abbb9f17b35e5548f7c8df710f084657b6f927b0b08b61b0e0510fb> (26) - *A* contrib/format-daffodil/src/main/resources/drill-module.conf <https://github.com/apache/drill/pull/2836/files#diff-ba82af9174d100bf1b07cc20c85feeb09fcdaee7aa291610ff5aaf1f13ddd8ee> (25) - *A* contrib/format-daffodil/src/test/java/org/apache/drill/exec/store/daffodil/TestDaffodilReader.java <https://github.com/apache/drill/pull/2836/files#diff-bd36b038725e2984107a508da683b62ccf1ad377934672d5b9f82df2838a3c9d> (652) - *A* contrib/format-daffodil/src/test/java/org/apache/drill/exec/store/daffodil/TestDaffodilUtils.java <https://github.com/apache/drill/pull/2836/files#diff-3067a2fb6fb2eb0149048c6d60bd154432f26bcd67f0e9e19e2c69ec9566ae1d> (44) - *A* contrib/format-daffodil/src/test/java/org/apache/drill/exec/store/daffodil/xsd/TestDaffodilSchema.java <https://github.com/apache/drill/pull/2836/files#diff-bde971a6448d3594674311523df7f179d4e3e598a6c94758a69737c62e9630e6> (79) - *A* contrib/format-daffodil/src/test/resources/data/simple.dat <https://github.com/apache/drill/pull/2836/files#diff-8f3d5ce87505acb0884ee71a495544967a17108445a5aa8649ac7e411027a806> (0) - *A* contrib/format-daffodil/src/test/resources/schema/complex1.dfdl.xsd <https://github.com/apache/drill/pull/2836/files#diff-3a0cbaf7934039ff28bb5d28c1d8e708e47bf41e2b3fc03a430bf04ba5342a4a> (46) - *A* contrib/format-daffodil/src/test/resources/schema/complexArray1.dfdl.xsd <https://github.com/apache/drill/pull/2836/files#diff-a090a4ba1ffd390f5c84267d29885aa306f8f71f2b34511e2c0e2f70a771be5a> (52) - *A* contrib/format-daffodil/src/test/resources/schema/simple.dfdl.xsd <https://github.com/apache/drill/pull/2836/files#diff-b668bf734e5dcadde495b70a7da22caef0e6d114446a94729f975a71b3b00e89> (39) - *M* contrib/pom.xml <https://github.com/apache/drill/pull/2836/files#diff-94fee37cfe3904e85b4b509ce36af377cd0852cc35959d1884e272bfd832ac60> (1) - *M* distribution/pom.xml <https://github.com/apache/drill/pull/2836/files#diff-ac19e9d7369629bf473b9b6c514012d5fecc319f98a6b9b57855f3202e970610> (5) - *M* distribution/src/assemble/component.xml <https://github.com/apache/drill/pull/2836/files#diff-d87f9550b7ca35bca5b5d29bb36d32a62f79eebe24cd123e5f87100e97b4aa6e> (1) - *M* exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/framework/SchemaNegotiator.java <https://github.com/apache/drill/pull/2836/files#diff-5ee19634060e900e678f8c7385f349de765131ea92a5ead4cc906e0e3eb77ded> (2) - *M* pom.xml <https://github.com/apache/drill/pull/2836/files#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8> (6) Patch Links: - https://github.com/apache/drill/pull/2836.patch - https://github.com/apache/drill/pull/2836.diff — Reply to this email directly, view it on GitHub <https://github.com/apache/drill/pull/2836>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALUDA5YL52MFBZ26MWX4VLX7H7GRANCNFSM6AAAAAA576F7JY> . You are receiving this because you are subscribed to this thread.Message ID: <apache/drill/pull/2...@github.com>
