Skip to content

With the non-negative constraint, L0Learn is getting stuck at a just a few non-zero coefficients #69

Description

@baogorek

It doesn't look like this repo is maintained, but I'll leave this as documentation. This is a non-standard situation. The simulation below is using regression to get pseudo design-based survey weights. Thus there's no intercept and we want them all to be positive. The problem is also ill-conditioned, but it doesn't seem to give the unconstrained method any problem.

See the #USER NOTE. If you run it both ways, you'll see the issue.

library(L0Learn)

Q <- 5000  # number of targets/samples
N <- 50000  # number of households/features - There will be this many weights
set.seed(34543)

# Metric matrix M (shape Q x N) - underdetermined system
M <- matrix(rlnorm(Q * N, meanlog=1.5, sdlog=.25), nrow = Q, ncol = N)

# Let's make a true w so we know that the target is in the space spanned by w
w_true <- rlnorm(N, meanlog=2, sdlog=1)

# Target vector y
y <- as.numeric(M %*% w_true)

# Fit with L0 and non-negative constraints
fit <- L0Learn.fit(
    M,
    y,
    penalty = "L0",
    maxSuppSize = 3000,  # Number that can be non-negative, but it will be a grid 
    # USER NOTE: toggle the following off after running through this once
    #lows = 0.0,  # Add non-negative box constraint
    highs = Inf,
    intercept = FALSE,
)

options(max.print = 1000)
print(fit)  # USER NOTE: look at the number of non-zero weights


single_lambda=4e-15  # You can look at print(fit), but I choose a very small value
w_hat <- as.numeric(coef(fit, lambda=single_lambda, gamma=0))

sum(w_hat != 0)  # confirm

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions