RE: RDD functions using GUI

2017-04-18 Thread Ke Yang (Conan) 
Ping... wonder why there aren't any such drag-n-drop GUI tool for creating 
batch query scripts?
Thanks

From: Ke Yang (Conan)
Sent: Monday, April 17, 2017 5:31 PM
To: 'dev@spark.apache.org' <dev@spark.apache.org>
Subject: RDD functions using GUI

Hi,
  Are there drag and drop GUI (code-free) for RDD functions available? i.e. a 
GUI that generates code based on drag-n-drops?
http://spark.apache.org/docs/latest/programming-guide.html#resilient-distributed-datasets-rdds

thanks for brainstorming

RDD functions using GUI

2017-04-17 Thread Ke Yang (Conan) 
Hi,
  Are there drag and drop GUI (code-free) for RDD functions available? i.e. a 
GUI that generates code based on drag-n-drops?
http://spark.apache.org/docs/latest/programming-guide.html#resilient-distributed-datasets-rdds

thanks for brainstorming

OOM by huge header size attack: setResponseHeaderSize won't work

2017-03-23 Thread Ke Yang (Conan) 
Folks,
  We use NiFi which embeds Jetty Server. Our test team found a security bug by 
intercepting the http request and replacing the header with a huge (say 1GB) 
text, which sent the response to NCM, which got OOM:
2017-03-07 03:44:03,522 WARN [NiFi Web Server-22] 
o.a.n.c.m.impl.HttpRequestReplicatorImpl Node request for 
[id=99a65e79-b856-4e43-9056-1451714498fc, apiAddress=129.188.35.109, 
apiPort=38484, socketAddress=129.188.35.109, socketPort=39494, 
siteToSiteAddress=129.188.35.109, siteToSitePort=null] encountered exception: 
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap 
space

 We tried setResponseHeaderSize here 
http://download.eclipse.org/jetty/stable-9/apidocs/org/eclipse/jetty/server/HttpConfiguration.html#setResponseHeaderSize-int-
 but it didn't seem to work: it seems to us that the huge fake header got 
received before this limit takes effect, as a result, the NCM got OOM in the 
first place.

Are we missing anything, or is there a potential bug with 
setResponseHeaderSize, please?

Btw, we also wonder if below workaround would work?

1.  Increase Nifi bootstrap.conf JVM heap size such as xmx8g

2.  Set connection timeout 
http://apache-nifi-developer-list.39713.n7.nabble.com/nifi-Cluster-setup-issue-td2988.html
nifi.cluster.manager.node.api.connection.timeout=30 sec
nifi.cluster.manager.node.api.read.timeout=30 sec


Thanks,
Conan