Hi
The version of solr in cdh is solr-4.10.3-cdh5.4.0 And the files in flume lib directory; [gbbdap36]/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib$ls activation-1.1.jar hive-metastore.jar ant-1.8.1.jar hive-serde.jar ant-launcher-1.8.1.jar hive-shims-0.23.jar antlr-2.7.7.jar hive-shims-common.jar antlr-runtime-3.4.jar hive-shims.jar aopalliance-1.0.jar hive-shims-scheduler.jar apacheds-i18n-2.0.0-M15.jar htrace-core-3.0.4.jar apacheds-kerberos-codec-2.0.0-M15.jar httpclient-4.2.5.jar apache-log4j-extras-1.1.jar httpcore-4.2.5.jar api-asn1-api-1.0.0-M20.jar irclib-1.10.jar api-util-1.0.0-M20.jar jackson-annotations-2.3.0.jar asm-3.2.jar jackson-core-2.3.1.jar async-1.4.0.jar jackson-core-asl-1.8.8.jar asynchbase-1.5.0.jar jackson-databind-2.3.1.jar avro-ipc.jar jackson-jaxrs-1.8.8.jar avro.jar jackson-mapper-asl-1.8.8.jar bonecp-0.7.1.RELEASE.jar jackson-xc-1.8.8.jar calcite-avatica-1.0.0-incubating.jar janino-2.7.6.jar calcite-core-1.0.0-incubating.jar jasper-compiler-5.5.23.jar calcite-linq4j-1.0.0-incubating.jar jasper-runtime-5.5.23.jar commons-beanutils-1.7.0.jar javax.inject-1.jar commons-beanutils-core-1.8.0.jar jaxb-api-2.2.2.jar commons-cli-1.2.jar jaxb-impl-2.2.3-1.jar commons-codec-1.8.jar jdo-api-3.0.1.jar commons-collections-3.2.1.jar jersey-client-1.9.jar commons-compiler-2.7.6.jar jersey-core-1.8.jar commons-compress-1.4.1.jar jersey-guice-1.9.jar commons-configuration-1.6.jar jersey-json-1.9.jar commons-dbcp-1.4.jar jersey-server-1.9.jar commons-digester-1.8.jar jersey-servlet-1.14.jar commons-el-1.0.jar jets3t-0.6.1.jar commons-httpclient-3.1.jar jettison-1.1.jar commons-io-2.4.jar jetty-6.1.26.cloudera.4.jar commons-jexl-2.1.1.jar jetty-util-6.1.26.cloudera.4.jar commons-lang-2.6.jar jline-2.11.jar commons-logging-1.1.3.jar joda-time-2.1.jar commons-math3-3.1.1.jar jopt-simple-3.2.jar commons-net-3.1.jar jsch-0.1.42.jar commons-pool-1.5.4.jar jsp-api-2.1.jar curator-client-2.6.0.jar jsr305-1.3.9.jar curator-framework-2.6.0.jar jta-1.1.jar curator-recipes-2.6.0.jar kafka_2.10-0.8.1.1.jar datanucleus-api-jdo-3.2.1.jar kite-data-core.jar datanucleus-core-3.2.2.jar kite-data-hbase.jar datanucleus-rdbms-3.2.1.jar kite-data-hive.jar derby-10.8.2.2.jar kite-hadoop-compatibility.jar eigenbase-properties-1.1.4.jar leveldbjni-all-1.8.jar flume-avro-source-1.5.0-cdh5.4.0.jar libfb303-0.9.2.jar flume-dataset-sink-1.5.0-cdh5.4.0.jar libthrift-0.9.2.jar flume-file-channel-1.5.0-cdh5.4.0.jar log4j-1.2.17.jar flume-hdfs-sink-1.5.0-cdh5.4.0.jar logredactor-1.0.2.jar flume-hive-sink-1.5.0-cdh5.4.0.jar mapdb-0.9.9.jar flume-irc-sink-1.5.0-cdh5.4.0.jar metrics-core-2.2.0.jar flume-jdbc-channel-1.5.0-cdh5.4.0.jar mina-core-2.0.4.jar flume-jms-source-1.5.0-cdh5.4.0.jar netty-3.6.2.Final.jar flume-kafka-channel-1.5.0-cdh5.4.0.jar opencsv-2.3.jar flume-kafka-source-1.5.0-cdh5.4.0.jar paranamer-2.3.jar flume-ng-auth-1.5.0-cdh5.4.0.jar parquet-avro.jar flume-ng-configuration-1.5.0-cdh5.4.0.jar parquet-column.jar flume-ng-core-1.5.0-cdh5.4.0.jar parquet-common.jar flume-ng-elasticsearch-sink-1.5.0-cdh5.4.0.jar parquet-encoding.jar flume-ng-embedded-agent-1.5.0-cdh5.4.0.jar parquet-format.jar flume-ng-hbase-sink-1.5.0-cdh5.4.0.jar parquet-generator.jar flume-ng-kafka-sink-1.5.0-cdh5.4.0.jar parquet-hadoop-bundle.jar flume-ng-log4jappender-1.5.0-cdh5.4.0.jar parquet-hadoop.jar flume-ng-morphline-solr-sink-1.5.0-cdh5.4.0.jar parquet-jackson.jar flume-ng-node-1.5.0-cdh5.4.0.jar pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar flume-ng-sdk-1.5.0-cdh5.4.0.jar protobuf-java-2.5.0.jar flume-scribe-source-1.5.0-cdh5.4.0.jar scala-library-2.10.1.jar flume-spillable-memory-channel-1.5.0-cdh5.4.0.jar serializer-2.7.1.jar flume-thrift-source-1.5.0-cdh5.4.0.jar servlet-api-2.5-20110124.jar flume-tools-1.5.0-cdh5.4.0.jar servlet-api-2.5.jar flume-twitter-source-1.5.0-cdh5.4.0.jar slf4j-api-1.7.5.jar groovy-all-2.1.6.jar slf4j-log4j12-1.7.5.jar gson-2.2.4.jar snappy-java-1.0.4.1.jar guava-11.0.2.jar ST4-4.0.4.jar guice-3.0.jar stax-api-1.0.1.jar guice-servlet-3.0.jar stax-api-1.0-2.jar hadoop-annotations.jar stringtemplate-3.2.1.jar hadoop-auth.jar twitter4j-core-3.0.3.jar hadoop-common.jar twitter4j-media-support-3.0.3.jar hadoop-yarn-api.jar twitter4j-stream-3.0.3.jar hadoop-yarn-common.jar velocity-1.7.jar hadoop-yarn-server-applicationhistoryservice.jar xalan-2.7.1.jar hadoop-yarn-server-common.jar xercesImpl-2.9.1.jar hadoop-yarn-server-resourcemanager.jar xml-apis-1.3.04.jar hadoop-yarn-server-web-proxy.jar xmlenc-0.52.jar hive-ant.jar xz-1.0.jar hive-common.jar zkclient-0.3.jar hive-exec.jar Thanks From: أنس الليثي [mailto:[email protected]] Sent: Wednesday, May 18, 2016 11:28 AM To: [email protected] Subject: Re: Flume MorphlineSolrSink I am not sure the issue you shared is the same issue you are facing. NoClassDefFoundError is totally different than NoSuchMethodError. I think solving this problem without upgrading is easy. Could you please tell what is the version of Solr you installed? and also put a list of the files in the flume lib directory. Best regards On 18 May 2016 at 11:11, Baris Akgun (Garanti Teknoloji) <[email protected]<mailto:[email protected]>> wrote: Hi You re right. I also find some jira for that problem. (https://issues.apache.org/jira/browse/FLUME-2392) The problem is solved in 1.6.0 version. Unfortunately, we are using oracle bda with cdh 5.4 ,therefore it is not easy to upgrade any service (flume, hive etc.) . We depended the oracle for upgrading. How can we solve that problem without upgrading the flume? It must be some other solutions. Thanks.. From: Gonzalo Herreros [mailto:[email protected]<mailto:[email protected]>] Sent: Wednesday, May 18, 2016 10:48 AM To: user Subject: Re: Flume MorphlineSolrSink Flume 1.5.0 is pretty old, Why don't you use version 1.6.0 included in CDH, that will ensure the library compatibility. On 18 May 2016 at 08:43, Baris Akgun (Garanti Teknoloji) <[email protected]<mailto:[email protected]>> wrote: Hi, I am trying to make real time indexing with using flume 1.5.0 and morphline solr sink. I am using packed base solr in my CDH. tier1.sinks = solrSink tier1.channels = channel1 # tier1.channels.channel1.type=memory tier1.channels.channel1.capacity=1000000 # tier1.sinks.solrSink.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink tier1.sinks.solrSink.batchSize = 100 tier1.sinks.solrSink.batchDurationMillis = 10000 tier1.sinks.solrSink.morphlineFile = morphlines.conf tier1.sinks.solrSink.morphlineId = morphline1 # tier1.sinks.solrSink.channel = channel1 The flume throw error in sink step. The error is given below. What is your ideas to solve the problem? Thanks. ERROR Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@748016d1 counterGroup:{ name:null counters:{} } } - Exception follows. java.lang.NoSuchMethodError: org.apache.lucene.util.fst.FST$BytesReader.skipBytes(J)V at org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:1117) at org.apache.lucene.util.fst.FST.readFirstRealTargetArc(FST.java:1030) at org.apache.lucene.util.fst.FST.readRootArcs(FST.java:487) at org.apache.lucene.util.fst.FST.cacheRootArcs(FST.java:475) at org.apache.lucene.util.fst.FST.finish(FST.java:458) at org.apache.lucene.util.fst.Builder.finish(Builder.java:464) at org.apache.lucene.analysis.synonym.SynonymMap$Builder.build(SynonymMap.java:284) at org.apache.lucene.analysis.synonym.FSTSynonymFilterFactory.loadSynonyms(FSTSynonymFilterFactory.java:145) at org.apache.lucene.analysis.synonym.FSTSynonymFilterFactory.inform(FSTSynonymFilterFactory.java:112) at org.apache.lucene.analysis.synonym.SynonymFilterFactory.inform(SynonymFilterFactory.java:89) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:675) at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:167) at org.apache.solr.schema.IndexSchemaFactory.create(IndexSchemaFactory.java:55) at org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:69) at org.kitesdk.morphline.solr.SolrLocator.getIndexSchema(SolrLocator.java:181) at org.kitesdk.morphline.solr.SanitizeUnknownSolrFieldsBuilder$SanitizeUnknownSolrFields.<init>(SanitizeUnknownSolrFieldsBuilder.java:70) at org.kitesdk.morphline.solr.SanitizeUnknownSolrFieldsBuilder.build(SanitizeUnknownSolrFieldsBuilder.java:52) at org.kitesdk.morphline.base.AbstractCommand.buildCommand(AbstractCommand.java:302) at org.kitesdk.morphline.base.AbstractCommand.buildCommandChain(AbstractCommand.java:249) at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:46) at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40) at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126) at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55) at org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101) at org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97) at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) at org.apache.flume.SinkRunner.start(SinkRunner.java:79) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Barış Akgün Analitik Veri Ambarı ve Büyük Veri Yönetimi Uzman Tel : Dahili : Faks : Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz. This message and attachments are confidential and intended solely for the individual(s) stated in this message. If you received this message although you are not the addressee, you are responsible to keep the message confidential. The sender has no responsibility for the accuracy or correctness of the information in the message and its attachments. Our company shall have no liability for any changes or late receiving, loss of integrity and confidentiality, viruses and any damages caused in anyway to your computer system. Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz. This message and attachments are confidential and intended solely for the individual(s) stated in this message. If you received this message although you are not the addressee, you are responsible to keep the message confidential. The sender has no responsibility for the accuracy or correctness of the information in the message and its attachments. Our company shall have no liability for any changes or late receiving, loss of integrity and confidentiality, viruses and any damages caused in anyway to your computer system. -- Anas Rabei Senior Software Developer Mubasher.info [email protected]<mailto:[email protected]> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz. This message and attachments are confidential and intended solely for the individual(s) stated in this message. If you received this message although you are not the addressee, you are responsible to keep the message confidential. The sender has no responsibility for the accuracy or correctness of the information in the message and its attachments. Our company shall have no liability for any changes or late receiving, loss of integrity and confidentiality, viruses and any damages caused in anyway to your computer system.
