Re: Review Request 45912: SAMZA-0.10.0: fix the bug of SamzaObjectMapper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45912/ --- (Updated April 8, 2016, 11:18 p.m.) Review request for samza. Repository: samza Description --- add test Diffs (updated) - samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 717b5dcad2aa22540deb08962bf2833e7dc5baa5 samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 4f1c14ce3838163c5af8c9d076238e0ed32619e1 Diff: https://reviews.apache.org/r/45912/diff/ Testing --- Thanks, Yuanchi Ning
Re: Review Request 45912: SAMZA-0.10.0: fix the bug of SamzaObjectMapper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45912/ --- (Updated April 8, 2016, 11:16 p.m.) Review request for samza. Repository: samza Description (updated) --- add test Diffs (updated) - docs/_config.yml a3bb8d48a31b5542b9a10dc3e5c741a244e0d77d docs/_docs/replace-versioned.sh d9fb383645aafc204493c94f2eac27b5fee7f786 docs/community/committers.md 4c8426323199e05acd250d65af1f652c41d5c127 docs/contribute/tests.md c4f992407344155378fa20ac4bcb431cb2147981 docs/learn/tutorials/versioned/deploy-samza-job-from-hdfs.md c2d934d7919c82f99ede1c4b12cb450439f99e47 docs/learn/tutorials/versioned/deploy-samza-to-CDH.md b5e0397219f00ecdc900957ce70323b943dd83de docs/learn/tutorials/versioned/index.md 8b2b69222eb6a89c514085934def9062cc63dc0c docs/learn/tutorials/versioned/remote-debugging-samza.md efef0455ed73b2db0de512879ed02a41a571b541 docs/learn/tutorials/versioned/run-in-multi-node-yarn.md 50ac2111d30cb0e9c1725da09e40c49d380a7b9a docs/learn/tutorials/versioned/upgrading-from-0.7.0-to-0.8.0.md 340762e71a07ecd531d071e43b180ed310ca92f1 docs/startup/download/index.md ee057140408abfd5ad896db0793d8627f6a3a10f docs/startup/hello-samza/versioned/index.md 8fb4a94484f662272231b8fbf65f007addee8189 gradle.properties b18c0cb62aec7592e8bfb1f2aa83c6b8eada867f samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 717b5dcad2aa22540deb08962bf2833e7dc5baa5 samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 4f1c14ce3838163c5af8c9d076238e0ed32619e1 Diff: https://reviews.apache.org/r/45912/diff/ Testing --- Thanks, Yuanchi Ning
Re: Jackson null pointer when upgrading to Samza 0.10.0
at. One difference though: the > minimum supported (i.e. tested YARN version) in Samza 0.10 is 2.6.1 and > LinkedIn is running 2.7.1. > > -Yi > > On Wed, Mar 23, 2016 at 3:15 PM, Yuanchi Ning <yuan...@uber.com> wrote: > > > But by in 0.10 release it's using 1.8.5 by default, you mean that other > > people are fine with this version when upgrading? > > > > On Wed, Mar 23, 2016 at 3:13 PM, Yuanchi Ning <yuan...@uber.com> wrote: > > > >> Hi Yi, > >> > >> This is aonther issue. I overwrote it in our rapper library called > >> athena-core-lib. The mvn dependency tree shows that it's picking up the > >> version 1.9.13, plus the command line that runs the program shows > 1.9.13. > >> We are currently testing on different Yarn versions to see if that's > >> related since our current version is 2.6.0. > >> > >> Thanks! > >> Yuanchi > >> > >> On Wed, Mar 23, 2016 at 2:57 PM, Yi Pan <nickpa...@gmail.com> wrote: > >> > >>> Hi, Yuanchi, > >>> > >>> Is this related w/ the issue you reported earlier regarding to "problem > >>> picking up checkpoint after upgrade" in another thread? I assume that > you > >>> are using the official Samza 0.10 release? That has jackson version > 1.8.5 > >>> by default. How do you change it in your own build/package to 1.9.13? > >>> > >>> Thanks! > >>> > >>> -Yi > >>> > >>> On Wed, Mar 23, 2016 at 12:00 PM, Yuanchi Ning < > ningyuanchi...@gmail.com > >>> > wrote: > >>> > >>>> Hi All, > >>>> > >>>> When we upgrade our existing samza job from 0.9 to 0.10, we found this > >>>> error below: > >>>> > >>>> > >>>> > /var/lib/hadoop-yarn/data/samza-yarn/usercache/yuanchi/appcache/application_1458330755534_0075/container_1458330755534_0075_02_01/__package/bin/run-class.sh: > >>>> line 63: OpenJDK: command not found > >>>> java version "1.7.0_95" > >>>> OpenJDK Runtime Environment (IcedTea 2.6.4) > >>>> (7u95-2.6.4-0ubuntu0.12.04.1) > >>>> OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) > >>>> Exception in thread "main" java.lang.NullPointerException > >>>> at java.io.StringReader.(StringReader.java:50) > >>>> at > >>>> > org.codehaus.jackson.JsonFactory.createJsonParser(JsonFactory.java:636) > >>>> at > >>>> > org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863) > >>>> at > >>>> > org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:71) > >>>> at > >>>> org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala) > >>>> > >>>> > >>>> And that line 71 in SamzaAppMaster is: > >>>> > >>>> val coordinatorSystemConfig = new > >>>> > >>>> > MapConfig(SamzaObjectMapper.getObjectMapper.readValue(System.getenv(ShellCommandConfig.ENV_COORDINATOR_SYSTEM_CONFIG), > >>>> classOf[Config])) > >>>> > >>>> And our checkpoint/ new version related properties are as below: > >>>> > >>>> # Job config # > >>>> > >>>> job.factory.class=org.apache.samza.job.yarn.YarnJobFactory > >>>> > >>>> job.name=trip-counter > >>>> > >>>> job.datacenter=sjc1 > >>>> > >>>> job.environment=sandbox > >>>> > >>>> job.coordinator.system=kafka > >>>> > >>>> job.coordinator.replication.factor=3 > >>>> > >>>> > >>>> # Task config # > >>>> > >>>> task.class=com.uber.athena.TripCounterTask > >>>> > >>>> task.inputs=kafka.trip_details,kafka.hp-api-client_signups > >>>> > >>>> task.outputTripTopic=trip_count_details > >>>> > >>>> task.outputClientSignUpsTopic=client_sign_ups_count_details > >>>> > >>>> > >>>> > task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory > >>>> > >>>> task.checkpoint.system=kafka > >>>> > >>>> task.checkpoint.replication.factor=3 > >>>> > >>>> > >>>> Plus our version for codehaus jackson dependencies is 1.9.13 as > >>>> suggested in other dev email threads. > >>>> > >>>> > >>>> Any ideas what would cause this issue? > >>>> > >>>> Thanks! > >>>> > >>>> Yuanchi > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Yuanchi Ning > >>>> > >>> > >>> > >> > > > -- Yuanchi Ning Master of Information Technology Very Large Information System School of Computer Science Carnegie Mellon University Mobile: (412)680-9774 Email: ningyuanchi...@gmail.com yuanc...@cs.cmu.edu yuanc...@andrew.cmu.edu
Re: Jackson null pointer when upgrading to Samza 0.10.0
Hi Yi, This is aonther issue. I overwrote it in our rapper library called athena-core-lib. The mvn dependency tree shows that it's picking up the version 1.9.13, plus the command line that runs the program shows 1.9.13. We are currently testing on different Yarn versions to see if that's related since our current version is 2.6.0. Thanks! Yuanchi On Wed, Mar 23, 2016 at 2:57 PM, Yi Pan <nickpa...@gmail.com> wrote: > Hi, Yuanchi, > > Is this related w/ the issue you reported earlier regarding to "problem > picking up checkpoint after upgrade" in another thread? I assume that you > are using the official Samza 0.10 release? That has jackson version 1.8.5 > by default. How do you change it in your own build/package to 1.9.13? > > Thanks! > > -Yi > > On Wed, Mar 23, 2016 at 12:00 PM, Yuanchi Ning <ningyuanchi...@gmail.com> > wrote: > >> Hi All, >> >> When we upgrade our existing samza job from 0.9 to 0.10, we found this >> error below: >> >> >> /var/lib/hadoop-yarn/data/samza-yarn/usercache/yuanchi/appcache/application_1458330755534_0075/container_1458330755534_0075_02_01/__package/bin/run-class.sh: >> line 63: OpenJDK: command not found >> java version "1.7.0_95" >> OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.12.04.1) >> OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) >> Exception in thread "main" java.lang.NullPointerException >> at java.io.StringReader.(StringReader.java:50) >> at >> org.codehaus.jackson.JsonFactory.createJsonParser(JsonFactory.java:636) >> at >> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863) >> at >> org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:71) >> at >> org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala) >> >> >> And that line 71 in SamzaAppMaster is: >> >> val coordinatorSystemConfig = new >> >> MapConfig(SamzaObjectMapper.getObjectMapper.readValue(System.getenv(ShellCommandConfig.ENV_COORDINATOR_SYSTEM_CONFIG), >> classOf[Config])) >> >> And our checkpoint/ new version related properties are as below: >> >> # Job config # >> >> job.factory.class=org.apache.samza.job.yarn.YarnJobFactory >> >> job.name=trip-counter >> >> job.datacenter=sjc1 >> >> job.environment=sandbox >> >> job.coordinator.system=kafka >> >> job.coordinator.replication.factor=3 >> >> >> # Task config # >> >> task.class=com.uber.athena.TripCounterTask >> >> task.inputs=kafka.trip_details,kafka.hp-api-client_signups >> >> task.outputTripTopic=trip_count_details >> >> task.outputClientSignUpsTopic=client_sign_ups_count_details >> >> >> task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory >> >> task.checkpoint.system=kafka >> >> task.checkpoint.replication.factor=3 >> >> >> Plus our version for codehaus jackson dependencies is 1.9.13 as >> suggested in other dev email threads. >> >> >> Any ideas what would cause this issue? >> >> Thanks! >> >> Yuanchi >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> Yuanchi Ning >> > >
Jackson null pointer when upgrading to Samza 0.10.0
Hi All, When we upgrade our existing samza job from 0.9 to 0.10, we found this error below: /var/lib/hadoop-yarn/data/samza-yarn/usercache/yuanchi/appcache/application_1458330755534_0075/container_1458330755534_0075_02_01/__package/bin/run-class.sh: line 63: OpenJDK: command not found java version "1.7.0_95" OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.12.04.1) OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) Exception in thread "main" java.lang.NullPointerException at java.io.StringReader.(StringReader.java:50) at org.codehaus.jackson.JsonFactory.createJsonParser(JsonFactory.java:636) at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863) at org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:71) at org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala) And that line 71 in SamzaAppMaster is: val coordinatorSystemConfig = new MapConfig(SamzaObjectMapper.getObjectMapper.readValue(System.getenv(ShellCommandConfig.ENV_COORDINATOR_SYSTEM_CONFIG), classOf[Config])) And our checkpoint/ new version related properties are as below: # Job config # job.factory.class=org.apache.samza.job.yarn.YarnJobFactory job.name=trip-counter job.datacenter=sjc1 job.environment=sandbox job.coordinator.system=kafka job.coordinator.replication.factor=3 # Task config # task.class=com.uber.athena.TripCounterTask task.inputs=kafka.trip_details,kafka.hp-api-client_signups task.outputTripTopic=trip_count_details task.outputClientSignUpsTopic=client_sign_ups_count_details task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory task.checkpoint.system=kafka task.checkpoint.replication.factor=3 Plus our version for codehaus jackson dependencies is 1.9.13 as suggested in other dev email threads. Any ideas what would cause this issue? Thanks! Yuanchi -- Yuanchi Ning
Picking up checkpoint when upgrade to 0.10.0 from 0.9?
Hi All, When we test upgrading our existing Samza job from 0.9 to 0.10, we saw our Kafka Lag metric (KafkaSystemConsumerMetrics "messages-behind-high-watermark ") kept zero. Since we stopped the old job for a while and then restart the job with 0.10 using the same name, the lag should at least spike at the beginning. In the application master we did see it's picking up the same checkpoint topic though. Any ideas? thanks! Yuanchi -- Yuanchi Ning
Samza Unit Test Instrucations
Hello Samza Team, This is Yuanchi Ning from Uber Data Engineering, Realtime Metrics, Streaming Platform team. We are planning to use Samza to process the realtime data we have, and thanks for developing such an awesome open source project. While I am building our streaming service using Samza, I am wondering is there anyway to do unit tests for each Samza application Task instead of integration test? Say, set up the mocker environment for feeding data into MessageEnvelope or something similar? Thanks for your assistance. Best, Yuanchi
