Re: Spark got incorrect scala version while using spark 3.2.1 and spark 3.2.2

[pengyh]

good answer. nice to know too.

Sean Owen wrote:

Spark is built with and ships with a copy of Scala. It doesn't use your
local version.


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

mod_perl and mod_python

[pengyh]

I know perl and python a bit well, most time use both of them for work.
besides mod_perl, there is also mod_python.
do you know what's the difference between them?
I never heard people using mod_python to make some jobs.

Thanks

Re: Experience running mod_perl2 with mpm_event on Solaris 11

[pengyh]

does perl5 support threading?

j...@sunstarsys.com wrote:

I discuss different Dynamic Programming Language thread support 
athttps://sunstarsys.com/CMS/technology.  The people in the Perl community at 
large who knock Perl5's ithreads support are doing nobody any favors.

Re: Data ingestion

[pengyh]

from my experience, spark can read/write from/to both mysql and hive 
fluently.


regards.


Akash Vellukai wrote:
How we could do data ingestion from MySQL to Hive with the help of Spark 
streaming and not with Kafka


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

shuffle a array

[pengyh]

in my modperl app I want to shuffle an array. such as this method in ruby,

irb(main):005:0> array
=> [1, 2, 3, 4]
irb(main):006:0> array.shuffle
=> [1, 4, 3, 2]


do you know how to achieve it in perl?

Thanks.

Re: Supported Hadoop versions for Spark 3.3

[pengyh]

my spark cluster can access either hadoop 2 or 3. so it doesn't care 
what the current hadoop version is.


Håkan Nordgren wrote:
Hi All: Which Hadoop versions (and distributions — Cloudera, 
Hortonworks, etc.) are supported for Spark 3.3 for the “Pre-built with 
user-provided Apache Hadoop” package from 
https://spark.apache.org/downloads.html? 



-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: fluentd with kafka 3 issues

[pengyh]

thank you. i will try fluent-bit for now.


Raphael Mazelier wrote:
maybe you can you try another log shipper to validate this assumption; 
there are plenty and fluentd is already an old (and buggy and slow one); 
you can try fluent-bit or vector maybe.

fluentd with kafka 3 issues

[pengyh]

do you konw if the current fluentd supports kafka 3 (my version is 3.2.1)?

fluentd dry-run outputs:

2022-08-12 10:47:24 +0800 [info]: gem 
'fluent-plugin-calyptia-monitoring' version '0.1.3'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-elasticsearch' 
version '5.2.3'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-flowcounter-simple' 
version '0.1.0'

2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-kafka' version '0.18.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-metrics-cmetrics' 
version '0.1.2'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-opensearch' version 
'1.0.7'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-prometheus' version 
'2.0.3'
2022-08-12 10:47:24 +0800 [info]: gem 
'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-record-modifier' 
version '2.1.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-rewrite-tag-filter' 
version '2.4.0'

2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-s3' version '1.7.1'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-systemd' version 
'1.0.5'

2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-td' version '1.2.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-webhdfs' version 
'1.5.0'

2022-08-12 10:47:24 +0800 [info]: gem 'fluentd' version '1.15.1'
2022-08-12 10:47:24 +0800 [info]: starting fluentd-1.15.1 as dry run 
mode ruby="2.7.6"

2022-08-12 10:47:24 +0800 [info]: brokers has been set: ["127.0.0.1:9092"]

my pipeline is: apache2 weblogs -> fluentd -> kafka -> spark

But the issue I met is I can't write the output to the correct topic 
from fluentd, the messages always fall into the default topic, not the 
one I specified.


The plugin I am using is:
https://github.com/fluent/fluent-plugin-kafka

which is built-in by td-agent (the latest version as well).

Thanks.

Re: is mpm_event safe for modperl handler? [EXT]

[pengyh]

thanks for letting me know this.


James Smith wrote:

mpm_event like all Apache mpm_s is much better at handling "issues" where nginx 
just goes F* and returns nothing. When we have issues we can detect them on apache mpms - 
but we often fail to see nginx errors as it sort of just dies!

Re: Unsubscribe

[pengyh]

to unsubscribe: user-unsubscr...@spark.apache.org


Shrikar archak wrote:



unsubscribe


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: is mpm_event safe for modperl handler? [EXT]

[pengyh]

I think running Ignix as front-end server and mod_perl for backend 
server is the more popular choice.





If you want the speed of mod_event for static content and the power of mod_perl 
for dynamic content - the best way is to run a lightweight mod_event apache in 
front of a mod_prefork to run the mod_perl

Re: [Spark SQL] Omit Create Table Statement in Spark Sql

[pengyh]

you have to saveAsTable or view to make a SQL query.


As the title, does Spark Sql have a feature like Flink Catalog to omit 
`Create Table` statement, and write sql query directly ?


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: is mpm_event safe for modperl handler?

[pengyh]

thanks. that does sound sorry.


No and neither is mod_worker. The only mpm you can safely use is 
prefork. This is, in my opinion, mod_perl's fatal flaw which will doom it.

Re: PerlAccessHandler for POST access

[pengyh]

LoadModule apreq_module modules/mod_apreq2.so

in your httpd.conf?


yes. as you see:
lrwxrwxrwx 1 root root 27 Aug  4 14:00 perl.load -> 
../mods-available/perl.load
lrwxrwxrwx 1 root root 29 Aug  4 14:01 apreq2.load -> 
../mods-available/apreq2.load





Have you tried it with mpm_prefork?


the development environment is mpm_prefork.



What version of Apache and mod_perl are you using exactly?


Apache/2.4.41 (Ubuntu) mod_apreq2-20101207/2.8.0 mod_perl/2.0.11 
Perl/v5.30.0 configured



ubuntu 20.04 x64 OS.

Thanks

Re: PerlAccessHandler for POST access

[pengyh]

multiple -d "x=y" should be working.




   -d "param1=value1=value2"

Re: PerlAccessHandler for POST access

[pengyh]

You need to share the complete GET & POST request with the data section.


OK as you can test this GET works:

http://fb.cloudcache.net/?timestamp=12345=906434463477769dba188a4b670ef425

but this POST doesn't work:

curl -X POST -d 'timestamp=12345' \
 -d 'authkey=906434463477769dba188a4b670ef425' \
 http://fb.cloudcache.net/


The server responds with:

Forbidden
You don't have permission to access this resource.


So how to fix it?

Thanks

is mpm_event safe for modperl handler?

[pengyh]

Hello

yes i know mpm_prefork is pretty good for modperl applications.
if I run modperl handler (for example, PerlAccessHandler) under 
mpm_event, is it safe for a production environment?


thanks.

Re: PerlAccessHandler for POST access

[pengyh]

Have you checked that the values of $key and $digest are equal?


as i have said, GET always works, but POST doesn't. so I am not sure 
where is wrong.

Re: PerlAccessHandler for POST access

[pengyh]

return $key eq $digest ? Apache2::Const::OK : Apache2::Const::FORBIDDEN;


there are no further customized code for Apache2::Const::OK.

the httpd.conf just as:


ServerAdmin webmaster@localhost
ServerName my.site.net
DocumentRoot /var/www/feedback

PerlPostConfigRequire /etc/apache2/modperl/startup.pl


SetHandler modperl
PerlAccessHandler MLFB



ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel warn
CustomLog ${APACHE_LOG_DIR}/mlfb.access.log combined

Re: PerlAccessHandler for POST access

[pengyh]

POST always gets a 403 error. but GET does get the correct response.

Thanks


What do you mean by “doesn’t work”.  Do you mean your code isn’t executed at 
all, or that it isn’t executing correctly?  It would be helpful to see the 
associated Apache config, as well.

PerlAccessHandler for POST access

[pengyh]

do you know for my this simple PerlAccessHandler, why HTTP GET works, 
but POST doesn't?


use strict;
use Apache2::RequestRec ();
use Apache2::RequestIO ();
use Apache2::Connection ();
use APR::Table ();
use Apache2::Const -compile => qw(OK FORBIDDEN);
use Apache2::Request;
use Digest::MD5 qw(md5_hex);


sub handler {

my $r = shift;
my $req = Apache2::Request->new($r);
my $ts = $req->param("timestamp");
my $key = $req->param("authkey");
my $digest = md5_hex($ts);

return $key eq $digest ? Apache2::Const::OK : 
Apache2::Const::FORBIDDEN;

}


1;


Thanks.

Re: Spark Scala API still not updated for 2.13 or it's a mistake?

[pengyh]

I can use scala 2.13 for spark-shell, but not spark-submit.

regards.

Spark 3.3.0 supports 2.13, though you need to build it for 2.13. The 
default binary distro uses 2.12.


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

log transfering into hadoop/spark

[pengyh]

since flume is not continued to develop.
what's the current opensource tool to transfer webserver logs into
hdfs/spark?

thank you.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[pengyh]

congrats!


Congrats Sophie! 

Re: Use case idea

[pengyh]

* streaming handler is still useful for spark, though there is flink as
alternative
* RDD is also useful for transform especially for non-structure data
* there are many SQL products in market like Drill/Impala, but spark is
more powerful for distributed deployment as far as I know
* we never used spark for AI training, but use keras/pytorch which are
pretty easy for development a model.


Perhaps you should try other systems in the market first, that will give
an unbiased view of databricks and SPARK being just over
glamourised tool. The hope of extending SPARK with a separate easy to
use query engine for deep learning and other AI systems is gone now with
Ray, SPARK community now just defends the lack of support, and direction
in this matter largely, which is a joke.



-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: unsubscribe

[pengyh]

you could be able to unsubscribe yourself by using the signature below.



To unsubscribe e-mail: user-unsubscr...@spark.apache.org


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Use case idea

[pengyh]

I don't think so. we were using spark integarted with Kafka for
streaming computing and realtime reports. that just works.



SPARK is now just an overhyped and overcomplicated ETL tool, nothing
more, there is another distributed AI called as Ray, which should be the
next billion dollar company instead of just building those features in
SPARK natively using a different computation engine :)


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Use case idea

[pengyh]

I am afraid the most sql functions spark has the other BI tools also have.

spark is used for high performance computing, not for SQL function
comparisoin.

Thanks.


In other terms: what analytics funcionality, that no One erp has, Spark offers ?


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: perl DBI n00b

[pengyh]

The doucument for DBI on CPAN is excellent.
Or you may look at this thread:
http://www.perlmonks.org/?node_id=22050


-Ursprüngliche Nachricht-
Von: Mark Haney ma...@abemblem.com
Gesendet: 2012-1-11 15:58:30
An: beginners@perl.org beginners@perl.org
Betreff: perl DBI n00b

I'm pretty new to the perl DBI world, so can someone point me to a good
tutorial/reference site or book or something? I'm very proficient with
databases and pretty good with basic perl, but not used together. Any
help would be appreciated.


--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/




___
SMS schreiben mit WEB.DE FreeMail - einfach, schnell und
kostenguenstig. Jetzt gleich testen! http://f.web.de/?mc=021192

--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/