goodseog opened a new issue, #30517:
URL: https://github.com/apache/airflow/issues/30517

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   I'm having issues with remote logging in Airflow version 2.5.1.
   I've been running various tests, and the one thing I'm currently wondering 
is the "--raw" option.
   
   In all task logs like,
   ```
   [2023-04-07, 11:33:08 KST] {standard_task_runner.py:82} INFO -  \
    Running: ['airflow', 'tasks', 'run', 'hourly_ml_log_compaction', 
'IsSuccess', \
    'scheduled__2023-04-07T01:20:00+00:00', '--job-id', '247669', '--raw', 
'--subdir', \
    'DAGS_FOLDER/.../dag_hdfs_compaction.py', '--cfg-path', '/tmp/tmp6lvshx_i']
   ```
   
   Also in ```BaseTaskRunner```
   ```python
           self._command = popen_prepend + self._task_instance.command_as_list(
               raw=True, # It gaves '--raw' option I guess
               pickle_id=local_task_job.pickle_id,
               mark_success=local_task_job.mark_success,
               job_id=local_task_job.id,
               pool=local_task_job.pool,
               cfg_path=cfg_path,
           )
   ```
   
   And I found on ```S3Taskhander```
   ```python
       def set_context(self, ti):
           super().set_context(ti)
           # Local location and remote location is needed to open and
           # upload local log file to S3 remote storage.
           self.log_relative_path = self._render_filename(ti, ti.try_number)
           self.upload_on_close = not ti.raw 
   ```
   
   As I understand it, if the "--raw" option is provided, 
   it is expected that ```upload_on_close == False``` and no logs will be 
uploaded to S3.
   
   
   \+
   
   In documentation, it seems "--raw" option has deprecated...
   ```
   usage: airflow tasks run [-h] [--cfg-path CFG_PATH] [-f] [-A] [-i] [-I] [-N] 
[-l] [--map-index MAP_INDEX] [-m] [-p PICKLE] [--pool POOL] [--ship-dag] [-S 
SUBDIR] [-v] dag_id task_id execution_date_or_run_id
   
   Run a single task instance
   
   positional arguments:
     dag_id                The id of the dag
     task_id               The id of the task
     execution_date_or_run_id
                           The execution_date of the DAG or run_id of the DAGRun
   
   optional arguments:
     -h, --help            show this help message and exit
     --cfg-path CFG_PATH   Path to config file to use instead of airflow.cfg
     -f, --force           Ignore previous task instance state, rerun 
regardless if task already succeeded/failed
     -A, --ignore-all-dependencies
                           Ignores all non-critical dependencies, including 
ignore_ti_state and ignore_task_deps
     -i, --ignore-dependencies
                           Ignore task-specific dependencies, e.g. upstream, 
depends_on_past, and retry delay dependencies
     -I, --ignore-depends-on-past
                           Ignore depends_on_past dependencies (but respect 
upstream dependencies)
     -N, --interactive     Do not capture standard output and error streams 
(useful for interactive debugging)
     -l, --local           Run the task using the LocalExecutor
     --map-index MAP_INDEX
                           Mapped task index
     -m, --mark-success    Mark jobs as succeeded without running them
     -p PICKLE, --pickle PICKLE
                           Serialized pickle object of the entire dag (used 
internally)
     --pool POOL           Resource pool to use
     --ship-dag            Pickles (serializes) the DAG and ships it to the 
worker
     -S SUBDIR, --subdir SUBDIR
                           File location or directory from which to look for 
the dag. Defaults to '[AIRFLOW_HOME]/dags' where [AIRFLOW_HOME] is the value 
you set for 'AIRFLOW_HOME' config you set in 'airflow.cfg' 
     -v, --verbose         Make logging output more verbose
   
   ```
   
   
   
   
   ### What you think should happen instead
   
   _No response_
   
   ### How to reproduce
   
   It is difficult to reproduce, but currently many tasks either upload logs to 
S3 or not.
   
   ### Operating System
   
   CentOS Linux
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon==7.1.0
   apache-airflow-providers-celery==3.1.0
   apache-airflow-providers-cncf-kubernetes==5.1.1
   apache-airflow-providers-common-sql==1.3.3
   apache-airflow-providers-ftp==3.3.0
   apache-airflow-providers-http==4.1.1
   apache-airflow-providers-imap==3.1.1
   apache-airflow-providers-jdbc==3.3.0
   apache-airflow-providers-mongo==3.1.1
   apache-airflow-providers-papermill==3.1.1
   apache-airflow-providers-postgres==5.4.0
   apache-airflow-providers-redis==3.1.0
   apache-airflow-providers-slack==7.2.0
   apache-airflow-providers-sqlite==3.3.1
   
   ### Deployment
   
   Other 3rd-party Helm chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to