e : " + sr.toString)
sr.toString
}
}
Thread.sleep(12)
try{
* val messagesJson = spark.read.json(messages) ===> getting NPE here
after restarting using WAL*
messagesJson.write.mode("append").parquet(data)
}
catch {
Hi Users,
I am trying to build fault tolerant spark solace consumer.
Issue :- we have to take restart of the job due to multiple issue load
average is one of them. At that time whatever spark is processing or
batches in the queue is lost. We can't replay it because we already had
send ack while
Hi Users,
Is there anyway through which I can monitor or raise alert if any flow file
got stuck in nifi queue.
For now operation team needs to manually check for these. If you can
suggest way through which I can achieve this that would be great.
Thanks,
Nayan
--
Thanks & Regards,
Nayan Sh
/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Thanks & Regards,
Nayan Sharma
*+91-8095382952*
<https://www.linkedin.com/in/nayan-sharma>
<http://stackoverflow.com/users/3687426/nayan-sharma?tab=profile>
NiFi 1.11 .0
On Thu, 13 Jan 2022 at 18:27, Juan Pablo Gardella <
gardellajuanpa...@gmail.com> wrote:
> Which Nifi version are you using?
>
> On Thu, 13 Jan 2022 at 09:14, nayan sharma
> wrote:
>
>> Hi All,
>>
>>
>>
>> We are using consumeJ
that high *CLIENT_CLIENT_UNBIND* requests from applications
and that leads to very high solace logging events.
Any suggestion would be appreciated.
Thanks & Regards,
Nayan Sharma
*+91-8095382952*
<https://www.linkedin.com/in/nayan-sharma>
<http://stackoverflow.com/users/3687426/nay
https://imgur.com/aCzkWfu
On 2020/07/30 17:32:47, nayan sharma wrote:
> Hi Users,
> I am using mergeContent for emitting flow files when size will be greater
> than 1 Gb. It is scheduled to run/check for files every 30sec. MergeContent
> has following configuration
>ht
Hi Users,
I am using mergeContent for emitting flow files when size will be greater than
1 Gb. It is scheduled to run/check for files every 30sec. MergeContent has
following configuration
https://imgur.com/undefined
but it doesn't wait for anything. It emits files as soon as it wakes. It is
Mark
>
> On Dec 10, 2019, at 11:13 AM, nayan sharma
> wrote:
>
> Hi Mark,
>
> I was trying to update NIFI cluster to 1.10.0 but I am getting error
> *Connection
> State changed to SUSPENDED,RECONNECTED and
> zookeeper.KeeperException$ConnectionLossException :
>
require zookeeper 3.5.5 ?
Thanks & Regards,
Nayan Sharma
*+91-8095382952*
<https://www.linkedin.com/in/nayan-sharma>
<http://stackoverflow.com/users/3687426/nayan-sharma?tab=profile>
On Wed, Dec 4, 2019 at 4:18 PM nayan sharma wrote:
> Hi Mark,
> One more thing I have not
Hi Mark,
One more thing I have noticed about the putHDFS that only two files has been
written and rest of them are in complete. Please refer image.
https://i.imgur.com/hu26hQG.png
Thank you,
Mark
On 2019/12/04 07:30:03, nayan sharma wrote:
> Hi Mark,
> Thanks for your valuable sugg
ompressContent and PutHDFS. Simply load balance the listing
> itself (which is very cheap because the FlowFiles have no content) and the
> data will automatically be balanced across the cluster.
>
> Thanks
> -Mark
>
>
> > On Dec 3, 2019, at 9:18 AM, nayan sharm
Hi,
I am using 2 nodes cluster.
nodes config Heap(max) 48gb & 64 core machine
Processor flow
ListSFTP--->FetchSFTP(all nodes with 10 threads)--->CompressContent(all
nodes,10 threads)-->PutHDFS
Queues shows it has 96gb in queue but when I do listing it shows no flow files.
Everything seems
arn user and my job is also running from the
same user.
Thanks,
Nayan
> On Mar 22, 2018, at 12:54 PM, Jorge Machado <jom...@me.com> wrote:
>
> Seems to me permissions problems ! Can you check your user / folder
> permissions ?
>
> Jorge Machado
>
>
>
>
Hi All,As druid uses Hadoop MapReduce to ingest batch data but I am trying spark for ingesting data into druid taking reference from https://github.com/metamx/druid-spark-batchBut we are stuck at the following error.Application Log:—>2018-03-20T07:54:28,782 INFO [task-runner-0-priority-0]
ames.zipWithIndex.view) {
> val data = firtRow(idx).asInstanceOf[String].split("\\^")
> var j = 0
> for(d<-data){
> schema = schema + colNames + j + ","
> j = j+1
> }
> }
> schema=schema.substring(0,schema.length-1)
>
t 3:29 AM, ayan guha <guha.a...@gmail.com> wrote:
>
> You are looking for explode function.
>
> On Mon, 17 Jul 2017 at 4:25 am, nayan sharma <nayansharm...@gmail.com
> <mailto:nayansharm...@gmail.com>> wrote:
> I’ve a Dataframe where in some columns there a
I’ve a Dataframe where in some columns there are multiple values, always
separated by ^
phone|contact|
ERN~58XX7~^EPN~5X551~|C~MXXX~MSO~^CAxxE~~3XXX5|
phone1|phone2|contact1|contact2|
ERN~5XXX7|EPN~5891551~|C~MXXXH~MSO~|CAxxE~~3XXX5|
How can this be achieved using loop
Hi All,
ERROR:-
Caused by: org.apache.spark.util.TaskCompletionListenerException: Connection
error (check network and/or proxy settings)- all nodes failed; tried
[[10.0.1.8*:9200, 10.0.1.**:9200, 10.0.1.***:9200]]
I am getting this error while trying to show the dataframe.
df.count =5190767
Test
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
isin query
> Date: 17 April 2017 at 8:13:24 PM IST
> To: nayan sharma <nayansharm...@gmail.com>, user@spark.apache.org
>
> How about using OR operator in filter?
>
> On Tue, 18 Apr 2017 at 12:35 am, nayan sharma <nayansharm...@gmail.com
> <mailto:nayansharm..
Dataframe (df) having column msrid(String) having values
m_123,m_111,m_145,m_098,m_666
I wanted to filter out rows which are having values m_123,m_111,m_145
df.filter($"msrid".isin("m_123","m_111","m_145")).count
count =0
while
df.filter($"msrid".isin("m_123")).count
count=121212
I have
Dataframe (df) having column msrid(String) having values
m_123,m_111,m_145,m_098,m_666
I wanted to filter out rows which are having values m_123,m_111,m_145
df.filter($"msrid".isin("m_123","m_111","m_145")).count
count =0
while
df.filter($"msrid".isin("m_123")).count
count=121212
I have
you
> please try that and let us know:
> Command:
> spark-submit --packages com.databricks:spark-csv_2.11:1.4.0
>
> On Fri, 7 Apr 2017 at 00:39 nayan sharma <nayansharm...@gmail.com
> <mailto:nayansharm...@gmail.com>> wrote:
> spark version 1.6.2
> scala versio
spark version 1.6.2
scala version 2.10.5
> On 06-Apr-2017, at 8:05 PM, Jörn Franke <jornfra...@gmail.com> wrote:
>
> And which version does your Spark cluster use?
>
> On 6. Apr 2017, at 16:11, nayan sharma <nayansharm...@gmail.com
> <mailto:nayansharm...@gmail.c
scalaVersion := “2.10.5"
> On 06-Apr-2017, at 7:35 PM, Jörn Franke <jornfra...@gmail.com> wrote:
>
> Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or the
> other way around?
>
> On 6. Apr 2017, at 15:54, nayan sharma <nayansharm...
In addition I am using spark version 1.6.2
Is there any chance of error coming because of Scala version or dependencies
are not matching.?I just guessed.
Thanks,
Nayan
> On 06-Apr-2017, at 7:16 PM, nayan sharma <nayansharm...@gmail.com> wrote:
>
> Hi Jorn,
> Thanks for repl
il.com> wrote:
>
> Is the library in your assembly jar?
>
> On 6. Apr 2017, at 15:06, nayan sharma <nayansharm...@gmail.com
> <mailto:nayansharm...@gmail.com>> wrote:
>
>> Hi All,
>> I am getting error while loading CSV file.
>>
>> v
Hi All,
I am getting error while loading CSV file.
val datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header",
"true").load("timeline.csv")
java.lang.NoSuchMethodError:
org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat;
I
Hi,
I wanted to skip all the headers of CSVs present in a directory.
After searching on Google I got to know that it can be done using
sc.wholetextfiles.
Can any one suggest me how to do that in Scala.?
Thanks & Regards,
Nayan Sh
Spark shutdown hook just wipes
> temp files
>
> On Thu, Mar 23, 2017 at 10:55 AM, Jörn Franke <jornfra...@gmail.com
> <mailto:jornfra...@gmail.com>> wrote:
> What do you mean by clear ? What is the use case?
>
> On 23 Mar 2017, at 10:16, nayan sharma <nayans
Does Spark clears the persisted RDD in case if the task fails ?
Regards,
Nayan
32 matches
Mail list logo