goodseog opened a new issue, #30517:
URL: https://github.com/apache/airflow/issues/30517
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
I'm having issues with remote logging in Airflow version 2.5.1.
I've been running various tests, and the one thing I'm currently wondering
is the "--raw" option.
In all task logs like,
```
[2023-04-07, 11:33:08 KST] {standard_task_runner.py:82} INFO - \
Running: ['airflow', 'tasks', 'run', 'hourly_ml_log_compaction',
'IsSuccess', \
'scheduled__2023-04-07T01:20:00+00:00', '--job-id', '247669', '--raw',
'--subdir', \
'DAGS_FOLDER/.../dag_hdfs_compaction.py', '--cfg-path', '/tmp/tmp6lvshx_i']
```
Also in ```BaseTaskRunner```
```python
self._command = popen_prepend + self._task_instance.command_as_list(
raw=True, # It gaves '--raw' option I guess
pickle_id=local_task_job.pickle_id,
mark_success=local_task_job.mark_success,
job_id=local_task_job.id,
pool=local_task_job.pool,
cfg_path=cfg_path,
)
```
And I found on ```S3Taskhander```
```python
def set_context(self, ti):
super().set_context(ti)
# Local location and remote location is needed to open and
# upload local log file to S3 remote storage.
self.log_relative_path = self._render_filename(ti, ti.try_number)
self.upload_on_close = not ti.raw
```
As I understand it, if the "--raw" option is provided,
it is expected that ```upload_on_close == False``` and no logs will be
uploaded to S3.
\+
In documentation, it seems "--raw" option has deprecated...
```
usage: airflow tasks run [-h] [--cfg-path CFG_PATH] [-f] [-A] [-i] [-I] [-N]
[-l] [--map-index MAP_INDEX] [-m] [-p PICKLE] [--pool POOL] [--ship-dag] [-S
SUBDIR] [-v] dag_id task_id execution_date_or_run_id
Run a single task instance
positional arguments:
dag_id The id of the dag
task_id The id of the task
execution_date_or_run_id
The execution_date of the DAG or run_id of the DAGRun
optional arguments:
-h, --help show this help message and exit
--cfg-path CFG_PATH Path to config file to use instead of airflow.cfg
-f, --force Ignore previous task instance state, rerun
regardless if task already succeeded/failed
-A, --ignore-all-dependencies
Ignores all non-critical dependencies, including
ignore_ti_state and ignore_task_deps
-i, --ignore-dependencies
Ignore task-specific dependencies, e.g. upstream,
depends_on_past, and retry delay dependencies
-I, --ignore-depends-on-past
Ignore depends_on_past dependencies (but respect
upstream dependencies)
-N, --interactive Do not capture standard output and error streams
(useful for interactive debugging)
-l, --local Run the task using the LocalExecutor
--map-index MAP_INDEX
Mapped task index
-m, --mark-success Mark jobs as succeeded without running them
-p PICKLE, --pickle PICKLE
Serialized pickle object of the entire dag (used
internally)
--pool POOL Resource pool to use
--ship-dag Pickles (serializes) the DAG and ships it to the
worker
-S SUBDIR, --subdir SUBDIR
File location or directory from which to look for
the dag. Defaults to '[AIRFLOW_HOME]/dags' where [AIRFLOW_HOME] is the value
you set for 'AIRFLOW_HOME' config you set in 'airflow.cfg'
-v, --verbose Make logging output more verbose
```
### What you think should happen instead
_No response_
### How to reproduce
It is difficult to reproduce, but currently many tasks either upload logs to
S3 or not.
### Operating System
CentOS Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==7.1.0
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-cncf-kubernetes==5.1.1
apache-airflow-providers-common-sql==1.3.3
apache-airflow-providers-ftp==3.3.0
apache-airflow-providers-http==4.1.1
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-jdbc==3.3.0
apache-airflow-providers-mongo==3.1.1
apache-airflow-providers-papermill==3.1.1
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-redis==3.1.0
apache-airflow-providers-slack==7.2.0
apache-airflow-providers-sqlite==3.3.1
### Deployment
Other 3rd-party Helm chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]