G-gen ã®ç岩ã§ããåœèšäºã§ã¯ Vertex AI Custom Training ã«ãã㊠ã«ã¹ã¿ã ã³ã³ãã ã䜿çšããæšæºã§ã¯æäŸãããŠããªã LightGBM ã¢ãã«ã®åŠç¿ãã å¯äžåºŠïŒSHAPïŒã®åºå ãŸã§å®è¡ããæ¹æ³ã玹ä»ããŸãã ã¯ããã« ãã«ãæžã¿ã³ã³ãããšã«ã¹ã¿ã ã³ã³ããã®äœ¿ãåã ã«ã¹ã¿ã ã³ã³ããã®å©ç¹ æ§æå³ åæèšå® ããŒã¿ã®æºåãšåå² ã«ã¹ã¿ã ã³ã³ããã®æºå ãã£ã¬ã¯ããªãšãªããžããªã®æºå åŠç¿ã¹ã¯ãªããã®äœæ Dockerfile ã®äœæ ã³ã³ããã®ãã«ããšããã·ã¥ åŠç¿ãžã§ãã®å®è¡ æšè«ãšè©äŸ¡ææšã®ç¢ºèª åæã¬ããŒãã®è§£é ã¯ããã« ãã«ãæžã¿ã³ã³ãããšã«ã¹ã¿ã ã³ã³ããã®äœ¿ãåã Vertex AI Custom Training ã«ã¯åŠç¿ãžã§ããå®è¡ããããã®ã³ã³ããã€ã¡ãŒãžãšããŠã倧ãã 2 ã€ã®éžæè¢ããããŸãã ã³ã³ããã®çš®é¡ ç¹åŸŽ åããŠããã±ãŒã¹ ãã«ãæžã¿ã³ã³ãã Google Cloud ãçšæããã€ã¡ãŒãž XGBoost ã TensorFlow ãªã©ãæšæºçãªãã¬ãŒã ã¯ãŒã¯ãããã«äœ¿ãããæ ã«ã¹ã¿ã ã³ã³ãã èªåã§ Dockerfile ãæžããŠäœæããã€ã¡ãŒãž LightGBM ãªã©æªæäŸã®ã©ã€ãã©ãªã䜿ãããæããç¬èªã®åŠçãçµã¿èŸŒã¿ããæ ã«ã¹ã¿ã ã³ã³ããã®å©ç¹ ãã«ãæžã¿ã³ã³ããã§ããžã§ãå®è¡æã®åŒæ°ã« requirements=["lightgbm", "shap"] ã®ããã«æå®ããããšã§ã©ã€ãã©ãªã远å ã§ããŸãããã«ãæžã¿ã³ã³ããã«ã€ããŠã¯ä»¥äžã®èšäºãåç
§ããŠãã ããã blog.g-gen.co.jp ãããå®åã®æ¬çªéçšã«ãããŠãå®è¡æã«ã©ã€ãã©ãªãåçã«ã€ã³ã¹ããŒã«ããããšã¯ã以äžã®ãã¡ãªããããããŸãã 1 ç¹ç®ã¯ã ç°å¢ã®åçŸæ§ãäœäžãã ããšã§ãã ãžã§ããå®è¡ãããã³ã«ã€ã³ã¿ãŒãããããææ°ã®ããã±ãŒãžãååŸãããããäŸåã©ã€ãã©ãªã®ããŒãžã§ã³ãäžãã£ãããã«çªç¶ãžã§ããèœã¡ãããåŠç¿çµæãå€ãã£ãŠããŸããšãã£ããæ¬çªéçšã§é¿ããããªã¹ã¯ãæããŸãã 2 ç¹ç®ã¯ã å®è¡ã®ãã³ã«ãªãŒããŒããããçºçãã ããšã§ãã æ¯åã©ã€ãã©ãªãããŠã³ããŒãããŠã€ã³ã¹ããŒã«ããåŠçãèµ°ããããäœèšãªåŸ
ã¡æéãçºçããŸãã ã«ã¹ã¿ã ã³ã³ãããå©çšããããšã«ãããäžèšã®ãã¡ãªãããåé¿ã§ããŸãã åè : ã«ã¹ã¿ã ã³ã³ããã®æŠèŠ æ§æå³ åœèšäºã§ç޹ä»ããæé ã«é¢ããæ§æå³ã¯ä»¥äžã®ãšããã§ããç°å¢æ§ç¯ã®è² è·ã軜æžããããããœãŒã¹ã³ãŒãã®äœæã Python å®è¡ç°å¢ã« Colab Enterprise ã䜿çšããŸãã åæèšå® ã¯ããã«ã©ã€ãã©ãªã®ã€ã³ã¹ããŒã«ãšç°å¢å€æ°ã®èšå®ãè¡ããŸããä»åã¯å¯èŠåãè§£éã®ããã®ã©ã€ãã©ãªïŒ seaborn ã shap ïŒã远å ããŸãã # å¿
èŠãªã©ã€ãã©ãªã®ã€ã³ã¹ããŒã« !pip install google-cloud-aiplatform lightgbm shap scikit-learn pandas seaborn matplotlib -q # ãããžã§ã¯ããšãªãŒãžã§ã³ã®èšå® # â» ãèªèº«ã®ç°å¢ã«åãããŠæžãæããŠãã ãã PROJECT_ID = "your-project-id" LOCATION = "asia-northeast1" # ãã±ãããšãã©ã«ãã®å®çŸ© ROOT_BUCKET = "gs://your-bucket" EXPERIMENT_NAME = "diamonds-lgbm-v1" WORK_DIR = f "{ROOT_BUCKET}/{EXPERIMENT_NAME}" # Vertex AI SDK ã®åæå from google.cloud import aiplatform aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=WORK_DIR) # ãã±ãããååšããªãå Žåã®ã¿äœæ !gsutil mb -l {LOCATION} {ROOT_BUCKET} ããŒã¿ã®æºåãšåå² ããŒã¿ã¯æ©æ¢°åŠç¿ãã¢ã§äœ¿çšããããã€ã€ã¢ã³ãã®äŸ¡æ ŒããŒã¿ã䜿çšããŸãããã®ããŒã¿ã¯ã«ã©ãããªã©ã®æ°å€ããŒã¿ããã«ãããè²ãšãã£ãã«ããŽãªå€æ°ãå«ã¿ãŸãã åŠç¿ããŒã¿ãšæšè«ããŒã¿ã«åå²ã㊠Cloud Storage ã«ä¿åããŸãã import seaborn as sns from sklearn.model_selection import train_test_split import pandas as pd # ããŒã¿ã®ããŒã (~54,000è¡) df = sns.load_dataset( 'diamonds' ) # æååã«ã©ã ã 'category' åã«å€æ cat_cols = [ 'cut' , 'color' , 'clarity' ] for col in cat_cols: df[col] = df[col].astype( 'category' ) # åŠç¿ããŒã¿ãšæšè«ããŒã¿ã« 90:10 ã®å²åã§åå² train_full_df, test_df = train_test_split(df, test_size= 0.1 , random_state= 42 ) # ããŒã¿ã®ä¿å train_filename = "train.csv" train_full_df.to_csv(train_filename, index= False ) test_filename = "test.csv" test_df.to_csv(test_filename, index= False ) # GCS ãžã¢ããããŒã !gsutil cp {train_filename} {WORK_DIR}/data/{train_filename} !gsutil cp {test_filename} {WORK_DIR}/data/{test_filename} print (f "åŠç¿ããŒã¿: {WORK_DIR}/data/{train_filename}" ) print (f "æšè«ããŒã¿: {WORK_DIR}/data/{test_filename}" ) ã«ã¹ã¿ã ã³ã³ããã®æºå ãã£ã¬ã¯ããªãšãªããžããªã®æºå Colab Enterprise äžã«äœæ¥ãã£ã¬ã¯ããªãçšæããGoogle Cloud äžã«å®æããã³ã³ããã®ä¿åå
ãšãªã Artifact Registry ã®ãªããžããªãäœæããŸãã # äœæ¥çšãã£ã¬ã¯ããªã®äœæ !mkdir -p custom_container # Artifact Registry ã«ãªããžããªãäœæ (ååã®ã¿) !gcloud artifacts repositories create custom-training-repo \ --repository- format =docker \ --location={LOCATION} \ --description= "Custom Training Repository" || true åŠç¿ã¹ã¯ãªããã®äœæ ã³ã³ããå
ã§å®è¡ããã task.py ãäœæããŸãã ä»åã¯ã¢ãã«ã®åŠç¿ã ãã§ãªããéåŠç¿ã確èªããããã®åŠç¿æ²ç·ãšãäºæž¬ã®æ ¹æ ã説æããããã®å¯äžåºŠã®ç»åãçæããã¢ãã«ãšäžç·ã« Cloud Storage ãžã¢ããããŒãããåŠçãçµã¿èŸŒã¿ãŸãã %%writefile custom_container/task.py import argparse import os import pandas as pd import lightgbm as lgb import shap import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from google.cloud import storage from urllib.parse import urlparse import warnings warnings.filterwarnings( 'ignore' ) parser = argparse.ArgumentParser() parser.add_argument( '--train-data-uri' , dest= 'train_data_uri' , type = str , required= True ) args = parser.parse_args() # --- GCS ããŠã³ããŒã / ã¢ããããŒãçšã®é¢æ° --- def download_from_gcs (gcs_uri, local_file): parsed_url = urlparse(gcs_uri) client = storage.Client() bucket = client.bucket(parsed_url.netloc) blob = bucket.blob(parsed_url.path.lstrip( "/" )) blob.download_to_filename(local_file) def upload_to_gcs (local_file, gcs_dir): parsed_url = urlparse(gcs_dir) client = storage.Client() bucket = client.bucket(parsed_url.netloc) blob_path = f "{parsed_url.path.lstrip('/').rstrip('/')}/{local_file}" bucket.blob(blob_path).upload_from_filename(local_file) # --- 1. ããŒã¿ã®æºå --- print (f "Downloading data from {args.train_data_uri}..." , flush= True ) local_train_file = "train.csv" download_from_gcs(args.train_data_uri, local_train_file) df = pd.read_csv(local_train_file) cat_cols = [ 'cut' , 'color' , 'clarity' ] for col in cat_cols: df[col] = df[col].astype( 'category' ) X = df.drop(columns=[ "price" ]) y = df[ "price" ] # ã¹ã¯ãªããå
ã§åŠç¿çšãšæ€èšŒçšã«åå² (ããŒã¿ãªãŒã¯é²æ¢) X_train, X_val, y_train, y_val = train_test_split(X, y, test_size= 0.1 , random_state= 42 ) # --- 2. ã¢ãã«ã®åŠç¿ --- print ( "Training LightGBM model..." , flush= True ) model = lgb.LGBMRegressor(n_estimators= 100 , random_state= 42 ) # åŠç¿éçšãèšé²ããããã« eval_set ãæž¡ã model.fit( X_train, y_train, eval_set=[(X_train, y_train), (X_val, y_val)], eval_names=[ 'train' , 'valid' ] ) # --- 3. åæç»åã®çæãšä¿å --- # â åŠç¿æ²ç·ã®æç» lgb.plot_metric(model, metric= 'l2' ) plt.title( 'Learning Curve (MSE)' ) plt.tight_layout() plt.savefig( "learning_curve.png" ) plt.close() # â¡ SHAPå€ïŒå¯äžåºŠïŒã®æç» print ( "Calculating SHAP values..." , flush= True ) explainer = shap.TreeExplainer(model) shap_values = explainer(X_val.sample( min ( 1000 , len (X_val)), random_state= 42 )) plt.figure() shap.plots.beeswarm(shap_values, show= False ) plt.title( "SHAP Feature Importance" ) plt.tight_layout() plt.savefig( "shap_importance.png" ) plt.close() # --- 4. ææç©ã®ã¢ããããŒã --- aip_model_dir = os.getenv( "AIP_MODEL_DIR" ) if aip_model_dir: print (f "Uploading artifacts to: {aip_model_dir}" , flush= True ) model.booster_.save_model( "model.txt" ) upload_to_gcs( "model.txt" , aip_model_dir) upload_to_gcs( "learning_curve.png" , aip_model_dir) upload_to_gcs( "shap_importance.png" , aip_model_dir) print ( "Upload completed." , flush= True ) Dockerfile ã®äœæ Dockerfile ãèšè¿°ããŸããããŒã¹ã€ã¡ãŒãžã«ã¯ Python 3.12 ãæå®ããLightGBM ã«å¿
èŠãª libgomp1 ãã€ã³ã¹ããŒã«ããŸãã %%writefile custom_container/Dockerfile FROM python: 3.12 -slim # LightGBM ã«å¿
é ã® OS ã©ã€ãã©ãªãã€ã³ã¹ããŒã« RUN apt-get update && apt-get install -y --no-install-recommends \ libgomp1 \ && rm -rf /var/lib/apt/lists/* # å¿
èŠãª Python ã©ã€ãã©ãªã®ã€ã³ã¹ããŒã« RUN pip install --no-cache- dir \ pandas scikit-learn lightgbm shap matplotlib google-cloud-storage WORKDIR /app COPY task.py /app/task.py ENTRYPOINT [ "python" , "task.py" ] ã³ã³ããã®ãã«ããšããã·ã¥ Cloud Build ã䜿çšããŠã³ã³ããããã«ãããããã·ã¥ããŸãã # Cloud Build ã§ãã«ããšããã·ã¥ãå®è¡ REPO_NAME = "custom-training-repo" IMAGE_URI = f "{LOCATION}-docker.pkg.dev/{PROJECT_ID}/{REPO_NAME}/lgbm-shap-trainer:latest" !gcloud builds submit --tag {IMAGE_URI} ./custom_container åŠç¿ãžã§ãã®å®è¡ äœæããèªäœã³ã³ãã ( IMAGE_URI ) ãæå®ããŠãåŠç¿ãžã§ããéä¿¡ããŸããåŒæ° base_output_dir ãæå®ããããšã§ãæå®ãã Cloud Storage ã®ãã¹é
äžã«ã¢ãã«ãç»åãä¿åã§ããŸãã # ãžã§ãã®å®çŸ© job = aiplatform.CustomContainerTrainingJob( display_name= "diamonds-lgbm-shap-job" , container_uri=IMAGE_URI, ) # ãžã§ãã®å®è¡ print ( "ãžã§ããéä¿¡ããŸãããå®äºãŸã§ãåŸ
ã¡ãã ãã..." ) job.run( machine_type= "n1-standard-4" , replica_count= 1 , args=[ f "--train-data-uri={WORK_DIR}/data/train.csv" ], # ææç©ã®ä¿åå
ãã©ã«ããæå® base_output_dir=f "{WORK_DIR}/model_output" ) æšè«ãšè©äŸ¡ææšã®ç¢ºèª ãžã§ãå®äºåŸãCloud Storage ããåŠç¿æžã¿ã¢ãã«ãããŠã³ããŒãããColab Enterprise äžã§ãã¹ãããŒã¿ã«å¯Ÿãã粟床è©äŸ¡ãè¡ããŸãã import numpy as np import lightgbm as lgb from sklearn.metrics import r2_score, mean_squared_error import pandas as pd # 1. åŠç¿ã®ææç©ã®ããŠã³ããŒã MODEL_DIR = f "{WORK_DIR}/model_output/model" print ( "åŠç¿æžã¿ã¢ãã«ãšåæç»åãããŠã³ããŒãããŸã..." ) !gsutil cp {MODEL_DIR}/model.txt . !gsutil cp {MODEL_DIR}/learning_curve.png . !gsutil cp {MODEL_DIR}/shap_importance.png . # 2. ãã¹ãããŒã¿ã®èªã¿èŸŒã¿ df_test = pd.read_csv(f "{WORK_DIR}/data/test.csv" ) cat_cols = [ 'cut' , 'color' , 'clarity' ] for col in cat_cols: df_test[col] = df_test[col].astype( 'category' ) X_test = df_test.drop(columns=[ "price" ]) y_true = df_test[ "price" ] # 3. ããŒã«ã«æšè«ã®å®è¡ local_model = lgb.Booster(model_file= "model.txt" ) predictions = local_model.predict(X_test) # 4. è©äŸ¡ææšã®èšç®ãšè¡šç€º r2 = r2_score(y_true, predictions) rmse = np.sqrt(mean_squared_error(y_true, predictions)) print ( "-" * 30 ) print (f "è©äŸ¡çµæ (ããŒã¿æ°: {len(y_true)}ä»¶)" ) print (f "R2 Score (決å®ä¿æ°): {r2:.4f}" ) print (f "RMSE (誀差ã®å€§ãã): {rmse:.4f}" ) print ( "-" * 30 ) 以äžã¯çè
ã®ç°å¢ã«ãããå®è¡çµæã§ããR2ã¹ã³ã¢ã 0.98 ãè¶
ãã粟床ã®é«ãã¢ãã«ãäœæã§ããŸããã ------------------------------ è©äŸ¡çµæ (ããŒã¿æ°: 5394ä»¶) R2 Score (決å®ä¿æ°): 0.9817 RMSE (誀差ã®å€§ãã): 543.6218 ------------------------------ åæã¬ããŒãã®è§£é åã«äºæž¬ç²ŸåºŠãåºãã ãã§ãªããAI ã ãªããã®äºæž¬ãããã®ã ãè§£éããããšã¯å®åã«ãããŠéèŠã§ããã³ã³ããå
ã§çæããåŠç¿æ²ç·ã®ç»åãš SHAP ãçšããåå¥ããŒã¿ã®åæçµæã確èªããŸãã import shap from IPython.display import Image, display print ( "=== åŠç¿æ²ç· (éåŠç¿ã®ç¢ºèª) ===" ) display(Image( "learning_curve.png" )) print ( " \n === å
šäœã®å¯äžåºŠ (SHAP Beeswarm) ===" ) display(Image( "shap_importance.png" )) # --- åå¥ããŒã¿ã«å¯ŸããSHAPïŒè¡šåœ¢åŒïŒ--- print ( " \n === ç¹å®ã®ããŒã¿ïŒ1ä»¶ç®ïŒã®äºæž¬ã®æ ¹æ ===" ) explainer = shap.TreeExplainer(local_model) single_instance = X_test.iloc[[ 0 ]] shap_values_single = explainer(single_instance) shap_df = pd.DataFrame({ "ç¹åŸŽé (Feature)" : single_instance.columns, "å®éã®å€ (Value)" : single_instance.values[ 0 ], "äŸ¡æ Œãžã®åœ±é¿ (SHAPå€)" : shap_values_single.values[ 0 ] }) shap_df = shap_df.reindex(shap_df[ "äŸ¡æ Œãžã®åœ±é¿ (SHAPå€)" ].abs().sort_values(ascending= False ).index) base_value = explainer.expected_value predicted_price = predictions[ 0 ] print (f "ãããŒã¹ã©ã€ã³äŸ¡æ Œ (å¹³å)ã: {base_value:.2f}" ) display(shap_df.style.format({ "äŸ¡æ Œãžã®åœ±é¿ (SHAPå€)" : "{:+.2f}" }).hide(axis= "index" )) print (f "ãæçµäºæž¬äŸ¡æ Œã: {predicted_price:.2f}" ) åŠç¿æ²ç·ïŒLearning CurveïŒ ã確èªãããšãåŠç¿ããŒã¿ãšæ€èšŒããŒã¿ã®èª€å·®ïŒMSEïŒãå
±ã«å³è©äžããã§åæããŠããŸãã ããã¯ãæªç¥ã®ããŒã¿ã§ããæ€èšŒããŒã¿ã«å¯ŸããŠãéåŠç¿ãèµ·ããããšãªãåŠç¿ãã§ããŠãã蚌æ ã§ãã å
šäœã®å¯äžåºŠ ã§ã¯ãäžã«ããç¹åŸŽéã»ã©äºæž¬ãžã®åœ±é¿åã倧ããããšã瀺ããŠããŸããæšªè»žã® 0 ãåºæºã«ãå³åŽã äŸ¡æ ŒãäžããèŠå ãå·ŠåŽã äŸ¡æ ŒãäžããèŠå ã§ãã ããããã®èµ€è²ã¯æ°å€ã倧ããããŒã¿ã§ãããéè²ã¯æ°å€ã®å°ããããŒã¿ã衚ããŠããŸããäŸãã° carat ã¯å³åŽã«èµ€è²ã§ãããããããŠããããã carat ã倧ããã»ã©é«äŸ¡ã«ãªã ããšãåãããŸãã æåŸã« ç¹å®ã®1ä»¶ã«å¯Ÿããäºæž¬ã®æ ¹æ ã衚圢åŒã§åºåããŸããã å
šäœã®å¹³åäŸ¡æ ŒïŒããŒã¹ã©ã€ã³ïŒãåºæºãšããŠããéãã0.24ã«ã©ãããšå°ãããããã€ãã¹è©äŸ¡ããéæåºŠïŒclarityïŒãVVS1ãšé«å質ã§ãããããã©ã¹è©äŸ¡ããšãã£ãããã«ãæçµçãªäºæž¬äŸ¡æ Œã«è³ããŸã§ã®å
éšã®èšç®ããžãã¯ãããžãã¹éšéã«èª¬æã§ããŸãã ç岩 è£è²Ž (èšäºäžèЧ) ã¯ã©ãŠããœãªã¥ãŒã·ã§ã³éš ã¯ã©ãŠããã£ãããããŒèª² åæå±±çåšäœã®ãšã³ãžãã¢ãèå³åéã¯AI/MLãGoogle Cloud Partner Top Engineer ã«éžåºïŒ2025 / 2026ïŒã