ognjenit created RANGER-4792:
--------------------------------
Summary: Fix issue with creating index and import data in
ElasticSearch as Audit database
Key: RANGER-4792
URL: https://issues.apache.org/jira/browse/RANGER-4792
Project: Ranger
Issue Type: Bug
Components: admin, audit
Affects Versions: 2.4.0
Environment: Container:
- Linux: Debian buster
- Java: openjdk- 11
- Tested on kubernetes and openshift on AWS/Azure and on-prem
Reporter: ognjenit
Hi all,
I apologize in advance if I haven't adjusted this issue properly.
Short description:
I have deployed Trino with ranger-trino-plugin and I wanted to use
ElasticSearch (7.10.2) as a place where I want to store the audit. When I
configured ranger-admin to use elasticsearch (audit_store=elasticsearch and all
other parameters audit_elasticsearch_*) I started getting errors in the
catalina.out: java.lang.NoSuchFieldError: LUCENE_8_5_1. As I increased the
version of Lucena, it was written in the logs that an even higher version was
needed. So in the end, I moved it to 8.11.3 and 8.4.0 for lucene-spatial since
it is the latest.
After it was fixed, I tried to use https for elasticsearch protocol
(audit_elasticsearch_protocol) however, it always showed that ranger-admin use
http instead of https. I show in code that audit_elasticsearch_protocol is not
configured well.
As soon as it done, ranger admin successfully created ES index. However, I need
to move from MiscUtil.toDate to MiscUtil.toLocalDate for evtTime "column" since
I was getting error: Error converting value to date. Value =
2024-05-13T13:08:47.905Z
As soon as I fixed it, I found an error in Trino that the plugin couldn't
insert data into elasticsearch. After I upgraded httpcomponents bug-fix
version, it's started inserting data.
I opened PR with the fix 2.4.0 version, do I need to do the same on the master
branch?
PR: https://github.com/apache/ranger/pull/314/files
h4. 1. Lucene version - fixed problem with writing data to ElasticSearch
{*}Error{*}: java.lang.NoSuchFieldError: LUCENE_8_5_1
I tried to change minor version one by one, but only latest version fit for me.
Changes:
* agents-audit/pom.xml: 311
* pom.xml: 241
h4. 2. Elastic search protocol - fixed problem with changing protocol
Even though I changed ranger.audit.elasticsearch.protocol from http to https,
audit plugin still using http protocol.
Changes:
* security-admin/scripts/ranger-admin-site-template.xml: 167-170
* security-admin/scripts/setup.sh: 79, 794-797
* security-admin/scripts/upgrade_admin.py: 116
* security-admin/src/main/resources/conf.dist/ranger-admin-site.xml: 53-57
*
security-admin/src/test/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsServiceTest.java:
56
h4. 3. Audit plugin - cannot write audit to ES
{*}Error{*}: bootstrap method initialization exception
After changing the version of httpcomponents I started seeing audit
Changes:
* pom.xml: 137, 138, 140
h4. 4. Ranger admin console - Audit show 1-1-1970
{*}Erro{*}: Error converting value to date. Value = 2024-05-13T13:08:47.905Z
Even though evtTime was ok in ElasticSearch, ranger couldn't show it on GUI.
Changes:
*
security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java:
260
*
security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java:
239
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
