I am sorry Baris, I could not find an exact solution. Just a little trick you can try, The exception say *there is not method named skipBytes in the org.apache.lucene.util.fst.**FST class*. Try to get the latest jar of apache lucene from mvnrepository.com and place it in the lib directory of flume. Hope that work fir you.
Best regards On 18 May 2016 at 11:55, Baris Akgun (Garanti Teknoloji) < [email protected]> wrote: > Hi > > > > > > The version of solr in cdh is *solr-4.10.3-cdh5.4.0* > > > > And the files in flume lib directory; > > > > > [gbbdap36]/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib$ls > > activation-1.1.jar hive-metastore.jar > > ant-1.8.1.jar hive-serde.jar > > ant-launcher-1.8.1.jar hive-shims-0.23.jar > > antlr-2.7.7.jar hive-shims-common.jar > > antlr-runtime-3.4.jar hive-shims.jar > > aopalliance-1.0.jar hive-shims-scheduler.jar > > apacheds-i18n-2.0.0-M15.jar htrace-core-3.0.4.jar > > apacheds-kerberos-codec-2.0.0-M15.jar httpclient-4.2.5.jar > > apache-log4j-extras-1.1.jar httpcore-4.2.5.jar > > api-asn1-api-1.0.0-M20.jar irclib-1.10.jar > > api-util-1.0.0-M20.jar > jackson-annotations-2.3.0.jar > > asm-3.2.jar jackson-core-2.3.1.jar > > async-1.4.0.jar > jackson-core-asl-1.8.8.jar > > asynchbase-1.5.0.jar > jackson-databind-2.3.1.jar > > avro-ipc.jar jackson-jaxrs-1.8.8.jar > > avro.jar > jackson-mapper-asl-1.8.8.jar > > bonecp-0.7.1.RELEASE.jar jackson-xc-1.8.8.jar > > calcite-avatica-1.0.0-incubating.jar janino-2.7.6.jar > > calcite-core-1.0.0-incubating.jar > jasper-compiler-5.5.23.jar > > calcite-linq4j-1.0.0-incubating.jar > jasper-runtime-5.5.23.jar > > commons-beanutils-1.7.0.jar javax.inject-1.jar > > commons-beanutils-core-1.8.0.jar jaxb-api-2.2.2.jar > > commons-cli-1.2.jar jaxb-impl-2.2.3-1.jar > > commons-codec-1.8.jar jdo-api-3.0.1.jar > > commons-collections-3.2.1.jar jersey-client-1.9.jar > > commons-compiler-2.7.6.jar jersey-core-1.8.jar > > commons-compress-1.4.1.jar jersey-guice-1.9.jar > > commons-configuration-1.6.jar jersey-json-1.9.jar > > commons-dbcp-1.4.jar jersey-server-1.9.jar > > commons-digester-1.8.jar jersey-servlet-1.14.jar > > commons-el-1.0.jar jets3t-0.6.1.jar > > commons-httpclient-3.1.jar jettison-1.1.jar > > commons-io-2.4.jar > jetty-6.1.26.cloudera.4.jar > > commons-jexl-2.1.1.jar > jetty-util-6.1.26.cloudera.4.jar > > commons-lang-2.6.jar jline-2.11.jar > > commons-logging-1.1.3.jar joda-time-2.1.jar > > commons-math3-3.1.1.jar jopt-simple-3.2.jar > > commons-net-3.1.jar jsch-0.1.42.jar > > commons-pool-1.5.4.jar jsp-api-2.1.jar > > curator-client-2.6.0.jar jsr305-1.3.9.jar > > curator-framework-2.6.0.jar jta-1.1.jar > > curator-recipes-2.6.0.jar kafka_2.10-0.8.1.1.jar > > datanucleus-api-jdo-3.2.1.jar kite-data-core.jar > > datanucleus-core-3.2.2.jar kite-data-hbase.jar > > datanucleus-rdbms-3.2.1.jar kite-data-hive.jar > > derby-10.8.2.2.jar > kite-hadoop-compatibility.jar > > eigenbase-properties-1.1.4.jar leveldbjni-all-1.8.jar > > flume-avro-source-1.5.0-cdh5.4.0.jar libfb303-0.9.2.jar > > flume-dataset-sink-1.5.0-cdh5.4.0.jar libthrift-0.9.2.jar > > flume-file-channel-1.5.0-cdh5.4.0.jar log4j-1.2.17.jar > > flume-hdfs-sink-1.5.0-cdh5.4.0.jar logredactor-1.0.2.jar > > flume-hive-sink-1.5.0-cdh5.4.0.jar mapdb-0.9.9.jar > > flume-irc-sink-1.5.0-cdh5.4.0.jar metrics-core-2.2.0.jar > > flume-jdbc-channel-1.5.0-cdh5.4.0.jar mina-core-2.0.4.jar > > flume-jms-source-1.5.0-cdh5.4.0.jar netty-3.6.2.Final.jar > > flume-kafka-channel-1.5.0-cdh5.4.0.jar opencsv-2.3.jar > > flume-kafka-source-1.5.0-cdh5.4.0.jar paranamer-2.3.jar > > flume-ng-auth-1.5.0-cdh5.4.0.jar parquet-avro.jar > > flume-ng-configuration-1.5.0-cdh5.4.0.jar parquet-column.jar > > flume-ng-core-1.5.0-cdh5.4.0.jar parquet-common.jar > > flume-ng-elasticsearch-sink-1.5.0-cdh5.4.0.jar parquet-encoding.jar > > flume-ng-embedded-agent-1.5.0-cdh5.4.0.jar parquet-format.jar > > flume-ng-hbase-sink-1.5.0-cdh5.4.0.jar parquet-generator.jar > > flume-ng-kafka-sink-1.5.0-cdh5.4.0.jar > parquet-hadoop-bundle.jar > > flume-ng-log4jappender-1.5.0-cdh5.4.0.jar parquet-hadoop.jar > > flume-ng-morphline-solr-sink-1.5.0-cdh5.4.0.jar parquet-jackson.jar > > flume-ng-node-1.5.0-cdh5.4.0.jar > pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar > > flume-ng-sdk-1.5.0-cdh5.4.0.jar protobuf-java-2.5.0.jar > > flume-scribe-source-1.5.0-cdh5.4.0.jar scala-library-2.10.1.jar > > flume-spillable-memory-channel-1.5.0-cdh5.4.0.jar serializer-2.7.1.jar > > flume-thrift-source-1.5.0-cdh5.4.0.jar > servlet-api-2.5-20110124.jar > > flume-tools-1.5.0-cdh5.4.0.jar servlet-api-2.5.jar > > flume-twitter-source-1.5.0-cdh5.4.0.jar slf4j-api-1.7.5.jar > > groovy-all-2.1.6.jar slf4j-log4j12-1.7.5.jar > > gson-2.2.4.jar snappy-java-1.0.4.1.jar > > guava-11.0.2.jar ST4-4.0.4.jar > > guice-3.0.jar stax-api-1.0.1.jar > > guice-servlet-3.0.jar stax-api-1.0-2.jar > > hadoop-annotations.jar stringtemplate-3.2.1.jar > > hadoop-auth.jar twitter4j-core-3.0.3.jar > > hadoop-common.jar > twitter4j-media-support-3.0.3.jar > > hadoop-yarn-api.jar > twitter4j-stream-3.0.3.jar > > hadoop-yarn-common.jar velocity-1.7.jar > > hadoop-yarn-server-applicationhistoryservice.jar xalan-2.7.1.jar > > hadoop-yarn-server-common.jar xercesImpl-2.9.1.jar > > hadoop-yarn-server-resourcemanager.jar xml-apis-1.3.04.jar > > hadoop-yarn-server-web-proxy.jar xmlenc-0.52.jar > > hive-ant.jar xz-1.0.jar > > hive-common.jar zkclient-0.3.jar > > hive-exec.jar > > > > > > Thanks > > > > *From:* أنس الليثي [mailto:[email protected]] > *Sent:* Wednesday, May 18, 2016 11:28 AM > *To:* [email protected] > *Subject:* Re: Flume MorphlineSolrSink > > > > I am not sure the issue you shared is the same issue you are facing. > > > > *NoClassDefFoundError* is totally different than *NoSuchMethodError*. > > > > I think solving this problem without upgrading is easy. > > > > Could you please tell what is the version of Solr you installed? and also > put a list of the files in the flume lib directory. > > > > Best regards > > > > On 18 May 2016 at 11:11, Baris Akgun (Garanti Teknoloji) < > [email protected]> wrote: > > Hi > > > > You re right. I also find some jira for that problem. ( > https://issues.apache.org/jira/browse/FLUME-2392) The problem is solved > in 1.6.0 version. > > > > Unfortunately, we are using oracle bda with cdh 5.4 ,therefore it is not > easy to upgrade any service (flume, hive etc.) . We depended the oracle for > upgrading. > > > > How can we solve that problem without upgrading the flume? It must be some > other solutions. > > > > Thanks.. > > > > *From:* Gonzalo Herreros [mailto:[email protected]] > *Sent:* Wednesday, May 18, 2016 10:48 AM > *To:* user > *Subject:* Re: Flume MorphlineSolrSink > > > > Flume 1.5.0 is pretty old, > > Why don't you use version 1.6.0 included in CDH, that will ensure the > library compatibility. > > > > On 18 May 2016 at 08:43, Baris Akgun (Garanti Teknoloji) < > [email protected]> wrote: > > Hi, > > > > I am trying to make real time indexing with using flume 1.5.0 and > morphline solr sink. I am using packed base solr in my CDH. > > > > > > tier1.sinks = solrSink > > tier1.channels = channel1 > > # > > tier1.channels.channel1.type=memory > > tier1.channels.channel1.capacity=1000000 > > # > > tier1.sinks.solrSink.type = > org.apache.flume.sink.solr.morphline.MorphlineSolrSink > > tier1.sinks.solrSink.batchSize = 100 > > tier1.sinks.solrSink.batchDurationMillis = 10000 > > tier1.sinks.solrSink.morphlineFile = morphlines.conf > > tier1.sinks.solrSink.morphlineId = morphline1 > > # > > tier1.sinks.solrSink.channel = channel1 > > > > > > The flume throw error in sink step. The error is given below. What is > your ideas to solve the problem? > > > > Thanks. > > > > > > ERROR > > Unable to start SinkRunner: { > policy:org.apache.flume.sink.DefaultSinkProcessor@748016d1 counterGroup:{ > name:null counters:{} } } - Exception follows. > > java.lang.NoSuchMethodError: > org.apache.lucene.util.fst.FST$BytesReader.skipBytes(J)V > > at org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:1117) > > at > org.apache.lucene.util.fst.FST.readFirstRealTargetArc(FST.java:1030) > > at org.apache.lucene.util.fst.FST.readRootArcs(FST.java:487) > > at org.apache.lucene.util.fst.FST.cacheRootArcs(FST.java:475) > > at org.apache.lucene.util.fst.FST.finish(FST.java:458) > > at org.apache.lucene.util.fst.Builder.finish(Builder.java:464) > > at > org.apache.lucene.analysis.synonym.SynonymMap$Builder.build(SynonymMap.java:284) > > at > org.apache.lucene.analysis.synonym.FSTSynonymFilterFactory.loadSynonyms(FSTSynonymFilterFactory.java:145) > > at > org.apache.lucene.analysis.synonym.FSTSynonymFilterFactory.inform(FSTSynonymFilterFactory.java:112) > > at > org.apache.lucene.analysis.synonym.SynonymFilterFactory.inform(SynonymFilterFactory.java:89) > > at > org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:675) > > at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:167) > > at > org.apache.solr.schema.IndexSchemaFactory.create(IndexSchemaFactory.java:55) > > at > org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:69) > > at > org.kitesdk.morphline.solr.SolrLocator.getIndexSchema(SolrLocator.java:181) > > at > org.kitesdk.morphline.solr.SanitizeUnknownSolrFieldsBuilder$SanitizeUnknownSolrFields.<init>(SanitizeUnknownSolrFieldsBuilder.java:70) > > at > org.kitesdk.morphline.solr.SanitizeUnknownSolrFieldsBuilder.build(SanitizeUnknownSolrFieldsBuilder.java:52) > > at > org.kitesdk.morphline.base.AbstractCommand.buildCommand(AbstractCommand.java:302) > > at > org.kitesdk.morphline.base.AbstractCommand.buildCommandChain(AbstractCommand.java:249) > > at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:46) > > at > org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40) > > at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126) > > at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55) > > at > org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101) > > at > org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97) > > at > org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) > > at org.apache.flume.SinkRunner.start(SinkRunner.java:79) > > at > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > > > > *Barış Akgün* > Analitik Veri Ambarı ve Büyük Veri Yönetimi > Uzman > > Tel > > : > > Dahili > > : > > Faks > > : > > > > Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir > ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis > olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune > uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde > yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da > sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin > ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, > butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve > bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz. > > This message and attachments are confidential and intended solely for the > individual(s) stated in this message. If you received this message although > you are not the addressee, you are responsible to keep the message > confidential. The sender has no responsibility for the accuracy or > correctness of the information in the message and its attachments. Our > company shall have no liability for any changes or late receiving, loss of > integrity and confidentiality, viruses and any damages caused in anyway to > your computer system. > > > > Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir > ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis > olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune > uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde > yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da > sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin > ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, > butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve > bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz. > > This message and attachments are confidential and intended solely for the > individual(s) stated in this message. If you received this message although > you are not the addressee, you are responsible to keep the message > confidential. The sender has no responsibility for the accuracy or > correctness of the information in the message and its attachments. Our > company shall have no liability for any changes or late receiving, loss of > integrity and confidentiality, viruses and any damages caused in anyway to > your computer system. > > > > > > -- > > Anas Rabei > Senior Software Developer > Mubasher.info > [email protected] > Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir > ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis > olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune > uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde > yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da > sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin > ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, > butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve > bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz. > > This message and attachments are confidential and intended solely for the > individual(s) stated in this message. If you received this message although > you are not the addressee, you are responsible to keep the message > confidential. The sender has no responsibility for the accuracy or > correctness of the information in the message and its attachments. Our > company shall have no liability for any changes or late receiving, loss of > integrity and confidentiality, viruses and any damages caused in anyway to > your computer system. > -- Anas Rabei Senior Software Developer Mubasher.info [email protected]
