Luận án Tiếp cận machine learning trong quản trị danh mục trên thị trường chứng khoán Việt Nam

83. Scott-Ram (2000), 'Managing Portfolio Risk for Periods of Stress', World Gold Council, Gold portfolio letter 11. 84. Sen J., S. Mondal, and G. Nath (2021), 'Robust Portfolio Design and Stock Price Prediction Using an Optimized LSTM Model', 2021 IEEE 18th India Council International Conference (INDICON), 1-6. 85. Serraino Gaia and Stanislav Uryasev (2013), 'Conditional Value-at-Risk (CVaR)', Encyclopedia of Operations Research and Management Science, Saul I. Gass và Michael C. Fu, Springer US, Boston, MA, 258-266. 86. Sharpe William F. (1963), 'A Simplified Model for Portfolio Analysis', Management Science, 9(2), 277-293. 87. Strong R.A. (2009), Portfolio construction, management and protection, 5th, South-Western, Cengage Learning. 88. Szczepocki Piotr (2019), 'Clustering companies listed on the Warsaw Stock Exchange according to time-varying beta', Econometrics, 23, 63-79. 89. Ta Bao Q., Vu T. Huynh, Khai Q. H. Nguyen, Phung N. Nguyen, and Binh H. Ho (2022), 'Maximal Predictability Portfolio Optimization Model and Applications to Vietnam Stock Market', Credible Asset Allocation, Optimal Transport Methods, and Related Topics, Cham, 559-578. 90. Ta Van-Dai, CHUAN-MING Liu, and Direselign Addis Tadesse (2020), 'Portfolio Optimization-Based Stock Prediction Using Long-Short Term Memory Network in Quantitative Trading', Applied Sciences, 10(2), 437. 91. Tibshirani Robert (1996), 'Regression Shrinkage and Selection via the Lasso', Journal of the Royal Statistical Society. Series B (Methodological), 58(1), 267- 288. 92. Tibshirani Robert, Guenther Walther, and Trevor Hastie (2001), 'Estimating the Number of Clusters in a Data Set Via the Gap Statistic', Journal of the Royal Statistical Society Series B, 63, 411-423. 124 93. Tibshirani Ryan and Larry Wasserman (2017), 'Sparsity, the lasso, and friends', Lecture notes from “Statistical Machine Learning,” Carnegie Mellon University, Spring. 94. Tikhonov Andrei Nikolaevich (1963), 'On the regularization of ill-posed problems', Doklady Akademii Nauk, 49-52. 95. Tobin J. (1958), 'Liquidity Preference as Behavior Towards Risk', The Review of Economic Studies, 25(2), 65-86. 96. Tom Mitchell (1997), 'Machine Learning', Publisher: McGraw Hill. 97. Treynor Jack (1965), 'How to rate management of investment funds'. 98. Truong Loc Dong, H. Swint Friday, and Tran My Ngo (2023), 'Market Reaction to Delisting Announcements in Frontier Markets: Evidence from the Vietnam Stock Market', Risks, 11(11), 201. 99. Võ Thị Quý and Nguyễn Minh Cao Hoàng (2011), 'Measuring Risk Tolerance And Establishing An Optimal Portfolio For Individual Investors In HOSE', Journal of Economic Development, June 2011, 33-38. 100. Wang Wuyu, Weizi Li, Ning Zhang, and Kecheng Liu (2020), 'Portfolio formation with preselection using deep learning from long-term financial data', Expert Systems with Applications, 143, 113042. 101. Ward Joe H. (1963), 'Hierarchical Grouping to Optimize an Objective Function', Journal of the American Statistical Association, 58(301), 236-244. 102. Warren Liao T. (2005), 'Clustering of time series data—a survey', Pattern Recognition, 38(11), 1857-1874. 103. Wu Fanyou, Rado Gazo, Eva Haviarova, and Bedrich Benes (2021), 'Wood identification based on longitudinal section images by using deep learning', Wood Science and Technology, 55, 1-11. 104. Yue Shihong, Xiuxiu Wang, and Miaomiao Wei (2008), 'Application of two- order difference to gap statistic', Transactions of Tianjin University, 14(3), 217- 221. 105. Zhang Yong, Yongbin Su, Weilong Liu, and Xingyu Yang (2022), 'Portfolio Optimization with Lstm-Based Return and Risk Information', Available at SSRN 4215299, Preprint. 106. Zhang Yuanyuan, Xiang Li, and Sini Guo (2018), 'Portfolio selection problems with Markowitz’s mean–variance framework: a review of literature', Fuzzy Optimization and Decision Making, 17(2), 125-158. 125 PHỤ LỤC Phụ lục 1. Code Python chạy kết quả thực nghiệm cho phương pháp Phân cụm chuỗi thời gian và Chính quy hóa Lasso # %% import numpy as np import pandas as pd import seaborn as sns import matplotlib.pyplot as plt import matplotlib.dates as mdates from datetime import date ### import riskfolio as rp ### from scipy.cluster.hierarchy import fcluster import scipy.cluster.hierarchy as hierarchy from scipy.spatial.distance import pdist ### from arch import arch_model ### import os import warnings warnings.filterwarnings("ignore") pd.options.display.float_format = '{:.5}'.format ### from cycler import cycler ### ## Hàm tính lợi suất def calculate_returns(df): return np.log1p(df.pct_change()).iloc[1:] # Hàm tính lợi suất tích lũy theo thời gian def calculate_portfolio_return(returns): portfolio_return = [0] for r in returns: value = portfolio_return[-1] + r portfolio_return.append(value) return portfolio_return[:-1] # Đặt thư mục mặc định os.chdir('D:\\ NCS\\Code_Final') # %% # Ngày bắt đầu và kết thúc giai đoạn first_day = '2013-01-02' last_day = '2019-12-31' # Các ngày không giao dịch dropdays = ['2018-01-23', '2018-01-24'] # Số ngày lịch sử - train N_hist = 496 # Số ngày kiểm tra - test N_test = 21 # Nhập dữ liệu dạng thô df_raw = pd.read_csv("Data\\HOSE_2008_2022_vnstock.csv") 126 df_raw["TradingDate"] = pd.to_datetime(df_raw["TradingDate"], format="%m/%d/%Y") ### df_raw_period = df_raw[(df_raw['TradingDate'] >= first_day) & (df_raw['TradingDate'] <= last_day)] df_raw_period = df_raw_period.set_index('TradingDate', drop=True) days_max = df_raw_period.groupby('Stock')['Close'].count().value_counts().idxmax() # %% df_raw_frame = df_raw_period.groupby('Stock')['Close'].apply(calculate_returns).to_frame() # Tính lợi suất các mã trong giai đoạn theo giá đóng cửa df_Returns = pd.pivot_table(df_raw_frame, values = 'Close', columns = 'TradingDate', index = 'Stock') df = df_Returns.T df = df.drop(dropdays, axis = 0) # %% # Chia các tập train - test def partition_data(df, n_train, n_test): train_set = [] test_set = [] i = 0 while(i <= df.shape[0] - n_train - 1): df_temp = df.iloc[i: i + min(n_train + n_test, df.shape[0] - i)] df_days_nan = df_temp.isna().sum(axis = 0) lst = np.where(df_days_nan == 0) df_temp2 = df_temp.iloc[:, np.array(lst)[0]] print(df_temp2.head()) column_list = [] for column in df_temp2.columns: value_counts = df_temp2[column].value_counts() column_list.append(column) df_temp3 = df_temp2[column_list] df_train_temp = df_temp3.iloc[: n_train] df_test_temp = df_temp3.iloc[n_train:] train_set.append(df_train_temp) test_set.append(df_test_temp) i += n_test return train_set, test_set # %% df_data = partition_data(df, N_hist, N_test) # %% # Nhập dữ liệu VN-Index df_VNIndex_raw = pd.read_csv("Data\VNIndex_2008_2022.csv") df_VNIndex_raw['TradingDate'] = pd.to_datetime(df_VNIndex_raw['TradingDate']) df_VNIndex_raw = df_VNIndex_raw.set_index("TradingDate") df_Returns_VNIndex = calculate_returns(df_VNIndex_raw) df_VNIndex_raw = df_VNIndex_raw.loc[first_day:last_day] # %% # Lãi suất phi rủi ro df_rf = pd.read_csv("Data/TraiPhieu10Nam.csv") df_rf["Date"] = pd.to_datetime(df_rf["Date"],format= "%d/%m/%Y") df_rf["Date"] = pd.to_datetime(df_rf["Date"],format= "%Y-%m-%d") 127 df_rf = df_rf.set_index("Date") df_rf = df_rf.sort_index() df_rf_last = df_rf['Last'] # %% # Tính lãi suất phi rủi ro đại diện trong giai đoạn def get_rf(returns): return df_rf_last.loc[returns.index[0]:returns.index[- 1]].mean()/(365*100) # def sharpe_ratio(returns): rf = get_rf(returns) return ((returns - rf).mean()) / ((returns - rf).std()) # %% def get_lowerret(returns): return df_Returns_VNIndex.loc[returns.index[0]:returns.index[-1]].mean() # %% def get_clusters(returns, label): stocks = returns.columns return [stocks[label == name] for name in np.unique(label)] %% def get_portfolio(cluster, returns, k = 2): # mỗi cụm lấy k asset có sharpe lớn nhất portflolio = [] for clus in cluster: portflolio += list((sharpe_ratio(returns[clus])).nlargest(k).index) return portflolio %% def optimize(df_returns): print(df_returns.index[-1]) # Building the portfolio object method_mu='hist' # Method to estimate expected returns method_cov='hist' # Method to estimate covariance matrix model='Classic' rm = 'MV' # Risk measure used, this time will be variance obj = 'MinRisk' # Objective function hist = True # Use historical scenarios for risk measures rf = get_rf(df_returns) # Risk free rate l = 0 # Risk aversion factor port = rp.Portfolio(returns = df_returns) lower_ret = df_Returns_VNIndex.loc[df_returns.index].mean().values[0] port.lowerret = lower_ret port.assets_stats(method_mu = method_mu, method_cov = method_cov, d = 0.94) # Estimate optimal portfolio: w = port.optimization(model = model, rm = rm, obj = obj, rf = rf, l = l, hist = hist) return w # %% def hierarchical_clustering(corr, linkage, max_k): # hierarchcial clustering # corr is matrix correlation dist = np.sqrt((1 - corr).round(5) / 2) dist = pd.DataFrame(dist, columns=corr.columns, index=corr.index) 128 p_dist = pdist(dist) clustering = hierarchy.linkage(p_dist, method=linkage) # cluster levels over from 1 to N-1 clusters cluster_lvls = pd.DataFrame(hierarchy.cut_tree(clustering), index=corr.columns) num_k = cluster_lvls.columns # save column with number of clusters cluster_lvls = cluster_lvls.iloc[:, ::-1] # reverse order to start with 1 cluster cluster_lvls.columns = num_k # set columns to number of cluster W_list = [] # get within-cluster dissimilarity for each k for k in range(min(len(cluster_lvls.columns), max_k)): level = cluster_lvls.iloc[:,k] # get k clusters D_list = [] # within-cluster distance list for i in range(np.max(level.unique())+1): cluster = level.loc[level == i] # cluster_dist = dist.loc[cluster.index, cluster.index] # get distance cluster_pdist = pdist(cluster_dist) # flatten and transform D = np.nan_to_num(cluster_pdist.std()) D_list.append(D) # append to list W_k = np.sum(D_list) W_list.append(W_k) W_list = pd.Series(W_list) n = corr.shape[0] limit_k = int(min(max_k, np.sqrt(n))) gaps = W_list.shift(-2) + W_list - 2*W_list.shift(-1) gaps = gaps[:limit_k] k = gaps.idxmax() + 2 k = max(k, 3) return clustering, k ### def fit_hierarchical(returns_set): N = len(returns_set) labels =[] corr_returns = [returns_set[i].corr() for i in range(N)] for i in range(N): Z, t = hierarchical_clustering(corr_returns[i], linkage = 'complete', max_k = 10) labels_h = fcluster(Z, t = t, criterion = 'maxclust') labels.append(labels_h) return labels # %% def plot_df(df_input, y_label = ''): ## Set the rcParams with color or linestyle plt.rc('axes', prop_cycle=(cycler('color', ['r', 'y', 'b', 'g', 'c', 'm']) + cycler('linestyle', ['-', '--', ':', '-.', '-', '--']))) fig, ax = plt.subplots(figsize=(16,8)) params = {'axes.labelsize': 20, 'axes.titlesize': 20, 'legend.fontsize': 18} ax.xaxis.set_major_locator(mdates.MonthLocator(bymonthday = 1, interval=3)) 129 ax.xaxis.set_major_formatter(mdates.DateFormatter('%b\n%Y')) plt.plot(df_input.index, df_input) plt.xlabel(xlabel='Trading Date') plt.ylabel(ylabel = y_label) plt.legend(df_input.columns) plt.tight_layout() plt.show() # %% def set_Full_Weights(df, w0): weights_Full = pd.DataFrame() weights_Full.index = df.columns weights_Full = weights_Full.join(w0) weights_Full = weights_Full.fillna(0) return weights_Full # %% def get_result_MV_NSS(df_returns): returns_train = df_returns[0] returns_test = df_returns[1] N = len(returns_train) weights_MV = [optimize(returns_train[i]) for i in range(N)] returns_port_MV = pd.concat([returns_test[i] @ weights_MV[i] for i in range(N)]) return returns_port_MV, weights_MV # %% df_returns_port_MV, df_w_MV = get_result_MV_NSS(df_data) # %% def get_result(df_returns, labels): returns_train = df_returns[0] returns_test = df_returns[1] N = len(returns_train) clusters = [get_clusters(returns_train[i], labels[i]) for i in range(N)] portfolios = [get_portfolio(clusters[i], returns_train[i]) for i in range(N)] weights = [optimize(returns_train[i][portfolios[i]]) for i in range(N)] df_returns_port = pd.concat([returns_test[i][portfolios[i]]@weights[i] for i in range(N)]) return df_returns_port, weights # %% df_returns_port_1TS , df_w_1TS = get_result(df_data, labels_1TS) # Các hàm với 2 chuỗi thời gian # %% def d_corr(x,y): return np.sqrt(2*(1-np.corrcoef(x, y)[0, 1])) %% def get_volatility_matrix(returns): vols = [] for col in returns.columns: model = arch_model(returns[[col]], mean='Zero', vol='GARCH', p=1, q=1) model_fitted = model.fit(disp = 'off') vols.append(model_fitted.conditional_volatility) return np.array(vols) %% 130 def distance_matrix(returns, vol_arr): returns_val = returns.values.T n_samples = returns_val.shape[0] X_distance = np.zeros((n_samples, n_samples)) for i in range(n_samples): for j in range(i+1, n_samples): d1 = d_corr(returns_val[i], returns_val[j]) d2 = d_corr(vol_arr[i], vol_arr[j]) if np.isnan(d2): print("there is a nan in d_corr of volalities") d2=0 d = 0.5*d1 + 0.5*d2 X_distance[i][j] = d X_distance[j][i] = d return X_distance # %% def hierarchical_clustering_MTS(returns, linkage, max_k): # corr is matrix correlation corr = returns.corr() volatility_matrices = get_volatility_matrix(returns) # tính distance_matrices distance_matrices = distance_matrix(returns, volatility_matrices) dist = pd.DataFrame(distance_matrices, columns=corr.columns, index=corr.index) p_dist = pdist(dist) clustering = hierarchy.linkage(p_dist, method=linkage) # cluster levels over from 1 to N-1 clusters cluster_lvls = pd.DataFrame(hierarchy.cut_tree(clustering), index=corr.columns) num_k = cluster_lvls.columns # save column with number of clusters cluster_lvls = cluster_lvls.iloc[:, ::-1] cluster_lvls.columns = num_k # set columns to number of cluster W_list = [] # get within-cluster dissimilarity for each k for k in range(min(len(cluster_lvls.columns), max_k)): level = cluster_lvls.iloc[:,k] # get k clusters D_list = [] # within-cluster distance list for i in range(np.max(level.unique())+1): cluster = level.loc[level == i] # cluster_dist = dist.loc[cluster.index, cluster.index] cluster_pdist = pdist(cluster_dist) D = np.nan_to_num(cluster_pdist.std()) D_list.append(D) # append to list W_k = np.sum(D_list) W_list.append(W_k) W_list = pd.Series(W_list) n = corr.shape[0] limit_k = int(min(max_k, np.sqrt(n))) gaps = W_list.shift(-2) + W_list - 2*W_list.shift(-1) gaps = gaps[:limit_k] k = gaps.idxmax() + 2 k = max(k, 3) return clustering, k # %% 131 def fit_hierarchical_MTS(returns_train): N = len(returns_train) labels =[] for i in range(N): print('Tap thu', i) Z, t = hierarchical_clustering_MTS(returns_train[i], linkage = 'complete', max_k=10) labels_h=fcluster(Z, t=t, criterion='maxclust') labels.append(labels_h) return labels labels_2TS = fit_hierarchical_MTS(df_data[0]) # %% df_returns_port_2TS, df_w_2TS = get_result(df_data, labels_2TS) # Sharpe ratio def sharpe(returns): rf = get_rf(returns) return ((returns - rf).mean() / (returns - rf).std()) # Value-at-risk def var(returns, alpha): sorted_returns = np.sort(returns) index = int(alpha * len(sorted_returns)) return abs(sorted_returns[index]) # Expected Shortfall (cVaR) def cvar(returns, alpha): sorted_returns = np.sort(returns) index = int(alpha * len(sorted_returns)) sum_var = sorted_returns[0] # Calculate the total VaR beyond alpha for i in range(1, index): sum_var += sorted_returns[i] return abs(sum_var / index) # Return the average VaR # MV cho phép bán khống def port_MV(df): rho_MV = float(df_Returns_VNIndex.loc[df.index[0]:df.index[- 1]].mean(0).iloc[0]) w_MV = cp.Variable(df.shape[1]) ACov = np.cov(df.T) p_cov = cp.psd_wrap(ACov) risk_MV = cp.quad_form(w_MV, p_cov) cons_MV = [cp.sum(w_MV) == 1, df.mean(0).to_numpy() @ w_MV == rho_MV] prob_MV = cp.Problem(cp.Minimize(risk_MV), cons_MV) prob_MV.solve() return w_MV.value ### def get_result_MV(df_returns): returns_train = df_returns[0] returns_test = df_returns[1] N = len(returns_train) weights = [port_MV(returns_train[i]) for i in range(N)] df_returns_port = pd.concat([returns_test[i] @ weights[i] for i in range(N)]) return df_returns_port, weights # %% df_returns_MV = get_result_MV(df_data)[0] # Các giá trị lựa chọn cho tham số phạt 132 gamma_vals = [10**(n) for n in np.arange(2, -5, -0.5)] #### LASSO N_Periods = 12 # Số lần thực hiện trong tập train ### def loss_fn(A, b, w): return cp.norm2(b - A @ w)**2 def regularizer(w): return cp.norm1(w) def objective_Lasso_fn(A, b, w, lambd): return loss_fn(A, b, w) + lambd * regularizer(w) ### Cross Validation def cross_val(df): N_train = df.shape[0] k_start = 0 k_end = N_train - N_Periods*N_test - 1 # df_std_port_AllSample = pd.DataFrame() while k_end < N_train - 1: A = df.iloc[k_start : k_end] n_A = A.shape[0] p = A.shape[1] A_test = df.iloc[k_end : k_end + N_test] # Tập test trong mẫu ## RVNIndex_Lasso = df_Returns_VNIndex.iloc[k_start : k_end] AveReturn_Lasso = RVNIndex_Lasso.mean() rho_insample = float(AveReturn_Lasso) b_insample = [rho_insample]*n_A w_insample = cp.Variable(p) gamma_insample = cp.Parameter(nonneg=True) objective_insample = objective_Lasso_fn(A.to_numpy(), np.transpose(b_insample), w_insample, gamma_insample) constraints_insample = [A.to_numpy().mean(0) @ w_insample == rho_insample, cp.sum(w_insample) == 1] lst_std_port = [] df_std_port_InSample = pd.DataFrame() for val in gamma_vals: prob_insample = cp.Problem(cp.Minimize(objective_insample), constraints_insample) gamma_insample.value = val prob_insample.solve(solver=cp.MOSEK) Return_test = A_test @ w_insample.value lst_std_port.append(Return_test.std()) df_std_port_InSample = pd.DataFrame(lst_std_port) df_std_port_AllSample = pd.concat([df_std_port_AllSample, df_std_port_InSample], axis=1, ignore_index=True) k_start += N_test k_end += N_test id_ld_std = df_std_port_AllSample.mean(1) id_ld = id_ld_std.argmin() return id_ld # %% def get_port_Lasso(df): N_train_sample, p = df.shape rho_sample = df_Returns_VNIndex.loc[df.index].mean(0) 133 b_sample = [rho_sample]*N_train_sample w_sample = cp.Variable(p) gamma_sample = cp.Parameter(nonneg=True) id_ld = cross_val(df) gamma_sample.value = gamma_vals[id_ld] objective_sample = objective_Lasso_fn(df.to_numpy(), np.array(b_sample).reshape(N_train_sample, ), w_sample, gamma_sample) constraints_sample = [df.to_numpy().mean(0) @ w_sample == rho_sample, cp.sum(w_sample) == 1] prob_sample = cp.Problem(cp.Minimize(objective_sample), constraints_sample) prob_sample.solve(solver=cp.MOSEK) return w_sample.value # %% def get_result_Lasso(df_returns): returns_train = df_returns[0] returns_test = df_returns[1] N = len(returns_train) weights = [get_port_Lasso(returns_train[i]) for i in range(N)] df_returns_port = pd.concat([returns_test[i] @ weights[i] for i in range(N)]) return df_returns_port, weights df_return_Lasso = get_result_Lasso(df_data)[0] 134 Phụ lục 2. Code Python chạy kết quả thực nghiệm cho phương pháp LSTM # %% LSTM days_max = df_raw_period.groupby('Stock')['Open'].count().value_counts().idxmax() lst_days = (df_raw_period.groupby('Stock')['Open'].count() == days_max) lst_stocks_fulldays = lst_days.index[np.where(lst_days == True)] dta_period = df_raw_period[df_raw_period['Stock'].isin(lst_stocks_fulldays)] # %% def SMA(df, k = 10): sma = df.rolling(k).mean() return sma ## def WMA(df, k=10): weights = np.arange(k, 0, step=-1) wma = df.rolling(k).apply(lambda x: np.dot(x, weights)/weights.sum(), raw=True) return wma # def computeRSI (data, time_window): diff = data.diff(1).dropna() up_chg = 0 * diff down_chg = 0 * diff up_chg[diff > 0] = diff[ diff>0 ] down_chg[diff < 0] = diff[ diff < 0 ] up_chg_avg = up_chg.ewm(com=time_window-1, min_periods=time_window).mean() down_chg_avg = down_chg.ewm(com=time_window-1 , min_periods=time_window).mean() rs = np.abs(up_chg_avg/down_chg_avg) rsi = 100 - 100/(1+rs) return rsi ### def stochastic(data, k_window, d_window, window): min_val = data.rolling(window=window, center=False).min() max_val = data.rolling(window=window, center=False).max() stoch = ( (data - min_val) / (max_val - min_val) ) * 100 K = stoch.rolling(window=k_window, center=False).mean() D = K.rolling(window=d_window, center=False).mean() return K, D ### from sklearn.preprocessing import StandardScaler, MinMaxScaler scaler = StandardScaler() scaler_target = MinMaxScaler() ######## def df_to_windowed_df(df_input): ### Tạo dữ liệu đầu vào cho LSTM Y = df_input['Close'] Y_returns = calculate_returns(Y) X_history = df_input.copy() df_SMA = SMA(df_input['Close'], k = 10) df_WMA = WMA(df_input['Close'], k = 10) df_RSI = computeRSI(df_input['Close'], 14) 135 df_K, df_D = stochastic(df_input['Close'], 3, 3, 14) ### df_K = df_K.dropna() df_D = df_D.dropna() dates = df_D.index Y_returns = Y_returns[dates] df_K = df_K[dates] X_history = X_history.loc[dates] df_SMA = df_SMA[dates] df_WMA = df_WMA[dates] df_RSI = df_RSI[dates] df_data = pd.concat([X_history, df_SMA, df_WMA, df_RSI, df_K, df_D, Y_returns], axis=1, ignore_index=True) df_data.dropna(inplace=True) return df_data # %% # Chuyển đổi dữ liệu thành dạng phù hợp cho mô hình: def create_dataset(dataset, lead=1): n_rows = dataset.shape[0] dates = dataset.index df_as_np = dataset.to_numpy() q_60 = int(len(dates) * .6) q_80 = int(len(dates) * .8) middle_matrix = df_as_np[:, 0:-1] middle_matrix_train, middle_matrix_val, middle_matrix_test = middle_matrix[:q_60], middle_matrix[q_60:q_80], middle_matrix[q_80:] # middle_matrix_train_scaled = scaler.fit_transform(middle_matrix_train) middle_matrix_val_scaled, middle_matrix_test_scaled = scaler.transform(middle_matrix_val), scaler.transform(middle_matrix_test) # target = df_as_np[:, -1] target_train, target_val, target_test = target[:q_60], target[q_60:q_80], target[q_80:] target_train_scaled = scaler_target.fit_transform(target_train.reshape(-1, 1)) target_val_scaled, target_test_scaled = scaler_target.transform(target_val.reshape(-1, 1)), scaler_target.transform(target_test.reshape(-1, 1)) # data_scaled = np.concatenate((middle_matrix_train_scaled, middle_matrix_val_scaled, middle_matrix_test_scaled), axis=0) target_scaled = np.concatenate((target_train_scaled, target_val_scaled, target_test_scaled), axis=0) X, y = [], [] for i in range(n_rows - lead): a = data_scaled[i:(i+lead), :] X.append(a) y.append(target_scaled[i+lead]) return dates, np.array(X), np.array(y) ### # %% from tensorflow.keras.models import Sequential from tensorflow.keras.optimizers import Adam 136 from tensorflow.keras import layers from keras.layers import Dense, LSTM from sklearn.model_selection import GridSearchCV from scikeras.wrappers import KerasClassifier, KerasRegressor from keras.callbacks import EarlyStopping from sklearn.metrics import mean_squared_error # %% def create_model(optimizer='adam', neurons=50): model = Sequential() model.add(LSTM(neurons, input_shape=(X_train.shape[1], X_train.shape[2]))) model.add(Dense(1)) model.compile(loss='mse', optimizer=optimizer) return model # %% # Tính lợi suất các mã trong giai đoạn theo giá đóng cửa df_returns_period = dta_period.groupby('Stock')['Close'].apply(calculate_returns) # Chuyển về dạng wide df_returns_period = df_returns_period.unstack().T # %% from sklearn import metrics from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score, explained_variance_score from sklearn.metrics import mean_absolute_percentage_error # %% Score_all = dict() df_all = pd.DataFrame() run_step = 1 ##### for stock_name in lst_stocks_fulldays: df_Stock = dta_period[dta_period['Stock'] == stock_name].copy() df_Stock = df_Stock.drop(['Stock'], axis=1) windowed_df = df_to_windowed_df(df_Stock) dates, X, y = create_dataset(windowed_df, lead = 10) # q_60 = int(len(dates) * .6) q_80 = int(len(dates) * .8) ### dates_train, X_train, y_train = dates[:q_60], X[:q_60], y[:q_60] dates_val, X_val, y_val = dates[q_60:q_80], X[q_60:q_80], y[q_60:q_80] dates_test, X_test, y_test = dates[q_80:], X[q_80:], y[q_80:] model = create_model() model.compile(loss='mse', optimizer=Adam(learning_rate=0.001), metrics=['mean_absolute_error']) es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10) history = model.fit(X_train, y_train, callbacks=[es], validation_data=(X_val, y_val), batch_size=32, epochs=100, verbose = 0) ### y_pred_norm = model.predict(X_test, batch_size = 4, verbose=0) y_pred_val = model.predict(X_val, batch_size = 4, verbose = 0) # Chuyển đổi ngược các giá trị dự đoán và giá trị thực tế về dạng gốc y_pred_orig = scaler_target.inverse_transform(y_pred_norm.reshape(-1, 1)).flatten() 137 y_test_orig = scaler_target.inverse_transform(y_test.reshape(-1, 1)).flatten() y_val_orig = scaler_target.inverse_transform(y_pred_val.reshape(-1, 1)).flatten() # Tính toán Mean Squared Error (MSE) mse = mean_squared_error(y_test, y_pred_norm) y_pred_valtest = y_val_orig.tolist() + y_pred_orig.tolist() df_all = pd.concat([df_all, pd.DataFrame({str(stock_name) : y_pred_valtest})], axis=1) # Tính toán độ lỗi trung bình tuyệt đối phần trăm (MAPE) mape = mean_absolute_percentage_error(y_test_orig, y_pred_orig) # Tính toán Mean Absolute Error (MAE) mae = mean_absolute_error(y_test_orig, y_pred_orig) # Tính toán Coefficient of determination (R2) r2 = r2_score(y_test_orig, y_pred_orig) # Tính toán Explained Variance Score (EVS) evs = explained_variance_score(y_test_orig, y_pred_orig) Score_all[stock_name] = [mape, mse, mae, r2, evs] # %% df_all.set_index(dates[q_60 + 10:], inplace = True) dates_train.shape, dates_val.shape, dates_test.shape # %% na_columns = np.where(df_all.isna().apply(sum, axis = 0) > 0) df_all.drop(columns = df_all.columns[na_columns], inplace = True) # %% df_pred = df_all.copy() df_real = df_returns_period.loc[df_all.index] df_real = df_real[df_pred.keys()] df_real_pred = df_real - df_pred df_real_test = df_real.loc[dates_test[0]:] df_pred_val = df_pred.loc[:dates_val[-1]] df_pred_test = df_pred.loc[dates_test[0]:] # %% df_rp_val = df_real_pred.loc[:dates_val[-1]] df_rp_test = df_real_pred.loc[dates_test[0]:] # %% def create_Vt(df_input): Tv = df_input.shape[0] Vt = np.zeros((df_input.shape[1], df_input.shape[1])) for i in range(Tv): Vt += np.outer(df_input.iloc[i], df_input.iloc[i].T) return (1/Tv)*Vt # %% import cvxpy as cp # %% ### Các tham số chung cho bài toán tối ưu method_mu='hist' # Method to estimate expected returns method_cov='hist' # Method to estimate covariance matrix model='Classic' rm = 'MV' # Risk measure used obj = 'MinRisk' # Objective function hist = True # Use historical scenarios for risk measures # %% 138 Returns_LSTM = [] Returns_MV = [] # weights_LSTM = [] weights_MV = {} N = df_pred_test.shape[1] Vt = create_Vt(df_rp_val) N_val = df_pred_val.shape[0] # for i in range(df_pred_test.shape[0]): rmax, rmin = max(df_pred_test.iloc[i]), min(df_pred_test.iloc[i]) r0 = 0.9*(rmax - rmin) + rmin lower_ret = df_RVNIndex.loc[:df_real_test.index[i]].tail(N_val).mean().values[0] w_LSTM = cp.Variable(N) Vt_edited = cp.psd_wrap(Vt) risk_LSTM = cp.quad_form(w_LSTM, Vt_edited) cons_LSTM = [w_LSTM >= 0, cp.sum(w_LSTM) == 1, w_LSTM @ cp.vec(df_pred_test.iloc[i]) >= r0] prob_LSTM = cp.Problem(cp.Minimize(risk_LSTM), cons_LSTM) prob_LSTM.solve(solver=cp.ECOS, verbose = False) Returns_LSTM.append(df_real_test.iloc[i] @ w_LSTM.value) print(df_pred_test.index[i]) Vt = (1/2)*(Vt + np.outer(df_rp_test.iloc[i], df_rp_test.iloc[i].T)) weights_LSTM.append(w_LSTM.value) #### MV dữ liệu lịch sử risk_free_rate = get_rf(df_pred_test.loc[:df_real_test.index[i]]) rf =risk_free_rate # Risk free rate print(rf) l = 0 # Risk aversion factor port_MV_model = rp.Portfolio(returns= df_real.loc[:df_real_test.index[i]].tail(N_val)) port_MV_model.lowerret = lower_ret port_MV_model.assets_stats(method_mu=method_mu, method_cov=method_cov, d=0.94) # Estimate optimal portfolio: w_MV = port_MV_model.optimization(model=model, rm=rm, obj=obj, rf=rf, l=l, hist=hist) Returns_MV.append(df_real_test.iloc[i] @ w_MV.values) weights_MV[i] = w_MV.values.flatten() # %% df_benchmark = pd.DataFrame() df_benchmark['MV_LSTM'] = Returns_LSTM df_benchmark['MV'] = np.array(Returns_MV).reshape(1, -1).flatten() df_benchmark['VNIndex'] = df_RVNIndex.loc[df_real_test.index].values df_benchmark.set_index(df_real_test.index, inplace = True)